Compare commits
19
Commits
@@ -774,3 +774,231 @@ Six live tests against fmr, all green, no device involved: `LiveServerTest`, `Li
|
||||
`LiveGranted`, `LiveDownstream`, `LiveUpload`, `LiveCompat`, `LiveEnrollment`. Plus 74 client unit
|
||||
tests and the full Go suite. Everything in the last several entries is verified from the PC; the
|
||||
app's UI (settings, history, deep-link enrollment) and `mtu.pmtud_up` remain device-only.
|
||||
|
||||
### throughput: a rate, plus the qualifier that makes it a measurement (server-v0.6.1 … v0.6.2)
|
||||
A throughput test reports the *smallest* limit on the path — and the sender's own ceiling is one of
|
||||
the candidates. If the server is asked for 50 Mbps and 50 Mbps arrives, the network was never the
|
||||
constraint and "50 Mbps" says nothing about it. So `perf.throughput_udp` always carries
|
||||
`limited_by` (duration | budget | rate | send_error) and `measures_network`, and a finding is
|
||||
raised only when the path is actually implicated. The live run against fmr reports 20 Mbit/s with
|
||||
`measures_network: false`, which is the correct and useful answer.
|
||||
|
||||
Loss is computed against the **sender's own count**, fetched from the observations API, not against
|
||||
the requested rate. A receiver alone cannot tell "the network dropped it" from "the sender never
|
||||
sent it", and guessing turns a healthy server-side limit into a phantom network fault. The server
|
||||
keeps one summary per action rather than per-packet records — a ten-second run at 50 Mbps is half a
|
||||
million packets, and a struct each would turn a measurement into memory exhaustion.
|
||||
|
||||
Sending is **paced**, on an absolute schedule. Unpaced would measure the server's NIC and the first
|
||||
queue it meets, then collapse into loss that reads as a network fault; sleep-per-packet would
|
||||
accumulate scheduler error and drift the rate down over a ten-second run.
|
||||
|
||||
Throughput gets its own grant budget sized from the request, so every *other* action stays bounded
|
||||
at 8 MiB. When the byte cap binds before the clock does, the **duration is shortened and reported**
|
||||
rather than the run being truncated: promising thirty seconds and delivering twenty-one is the same
|
||||
information with a surprise attached, and it keeps "the clock ended the run" as the normal case —
|
||||
the only case where the rate is a clean property of the path. That behaviour came out of a test
|
||||
that failed honestly (30 s at 100 Mbps needs 375 MB against a 256 MB cap).
|
||||
|
||||
It is **opt-in** in the run config, default off. A 5-second run at 50 Mbps moves ~30 MB; on a
|
||||
metered mobile connection that is the user's money, and a tool that spends it without being asked
|
||||
is not one people keep installed.
|
||||
|
||||
#### The bug the live test found
|
||||
The first live run delivered 104 packets and stopped after 50 ms. The grant's rate check exempted
|
||||
the first 50 ms entirely, meaning to be lenient at startup — the effect was the opposite. A sender
|
||||
could dump an unbounded burst into that free window, and the instant the check switched on it
|
||||
compared those bytes against 50 ms worth of allowance and refused everything until real time caught
|
||||
up. **Every short test passed** (downtrain sends 50 packets, big_send seven); every sustained send
|
||||
died fifty milliseconds in.
|
||||
|
||||
Replaced with a token bucket (`allowance = burst + rate × elapsed`), which is smooth from t=0.
|
||||
The burst is 100 ms of the allowed rate, floored at one ordinary datagram — deliberately one, since
|
||||
at 8 kbps a 64 KB floor is sixty-four seconds' worth, exactly the instant dump the ceiling exists to
|
||||
prevent. The pre-existing rate test caught that when I first tried the generous floor, and it was
|
||||
right to. Second half of the same bug: callers treated *any* refusal as terminal, so `TryAllow` now
|
||||
says why — a sender paces through a transient "too fast just now" and still stops dead on a spent
|
||||
budget or an expired grant. Both halves are pinned by regression tests.
|
||||
|
||||
### Findings registry (2026-08-01)
|
||||
Closes open item 1 of measurement-schema.md §9. A finding code is the stable, machine-readable half
|
||||
of a result — what a dashboard groups by and what someone greps a year of archived runs for — and
|
||||
that only holds if a code means exactly one thing forever. Ad-hoc string literals at fifteen call
|
||||
sites cannot promise that, and by the time the registry was written the failure had already
|
||||
happened.
|
||||
|
||||
**Two emitters had independently produced `connectivity.downstream_loss` and
|
||||
`connectivity.loss_downstream` for the same claim**, and nothing anywhere objected. Anyone
|
||||
aggregating either one would have silently seen half their data. Merged into
|
||||
`connectivity.loss_downstream`, paired with `loss_upstream` so the two directions read as a set.
|
||||
|
||||
**Two codes were also renamed out of `nat.*`.** `nat.udp_unreachable` is not about NAT — it means
|
||||
no replies came back — but the prefix determines the category, and the category determines which
|
||||
verdict light the finding rolls up into (§7.3). A `nat.*` code landing under *connectivity* is not
|
||||
a naming quibble; it changes which light turns red. Cheap to fix now, a breaking change later.
|
||||
|
||||
Codes are now declared as typed `FindingSpec`s carrying their category and default severity, and
|
||||
emitters reference the spec instead of retyping the string — so a typo is a compile error and two
|
||||
call sites cannot disagree about a finding's category.
|
||||
|
||||
`docs/findings-registry.md` is the contract, and a test reads it: it fails when the document and
|
||||
the registry have codes the other lacks, or when a severity differs. Documentation that drifts from
|
||||
its implementation is worse than none, because it still looks authoritative. The check scopes
|
||||
itself to table rows, so the prose can keep explaining which codes were retired and why.
|
||||
|
||||
Six tests: uniqueness, declared-vs-listed, prefix↔category agreement, naming convention, a
|
||||
word-order-anagram check (the shape the duplication actually took), and the document agreement.
|
||||
|
||||
### A real privacy leak, found by starting on the machine-readable schema (2026-08-01)
|
||||
The intent was `measurement.schema.json` (§8's promised companion). The first step — checking
|
||||
whether the anonymizer actually covers the fields the schema declares as sensitive — found that it
|
||||
did not, so that became the work.
|
||||
|
||||
**At the `balanced` level, five identifying values were being uploaded verbatim:**
|
||||
|
||||
| value | field | why it matters |
|
||||
|---|---|---|
|
||||
| `2001:…::150` | `networks[].link.addresses[].addr` | the device's own global IPv6 address — a strong, geolocatable device identifier |
|
||||
| `2a02:…::1` | `networks[].link.routes[].gateway` | identifies the ISP allocation |
|
||||
| `203.0.113.77` | `networks[].link.dns.servers[]` | the configured resolver |
|
||||
| `nas.example.lan` | `private_dns_hostname` | an internal hostname |
|
||||
| `example.lan` | `search_domains[]` | the internal domain |
|
||||
|
||||
The settings screen describes that level as pseudonymizing addresses. It was not.
|
||||
|
||||
**Root cause:** classification keyed on field *names*, and the schema's actual names (`addr`,
|
||||
`gateway`, `dst`, `servers`, `search_domains`, `private_dns_hostname`) had never been added to the
|
||||
table. Not a subtle bug — just an unfalsifiable design. The existing tests all passed, because each
|
||||
one checked a field somebody had remembered to write a case for.
|
||||
|
||||
**Two fixes, one of them structural:**
|
||||
1. The missing names were added.
|
||||
2. More importantly, a **shape-based backstop**: when a field name is unrecognised, the *value* is
|
||||
inspected, and anything shaped like an IPv4/IPv6 address or a MAC is treated as one. A name
|
||||
table can only protect fields someone thought of, which is precisely the wrong property for a
|
||||
privacy control. Hostnames are deliberately *not* inferred by shape — `train.udp_updown` is
|
||||
indistinguishable from a domain, and mangling a test type would corrupt the document to protect
|
||||
nothing.
|
||||
|
||||
`LeakTest` is the new guard and is written to fail for fields nobody has considered: it plants
|
||||
identifying values wherever one can actually occur and asserts none survive, rather than checking
|
||||
a list of known cases. It also pins that RFC1918 addresses still come through readable, so the
|
||||
test cannot pass by over-redacting everything.
|
||||
|
||||
Route prefixes and the unspecified address needed care in the transform: `0.0.0.0/0` and `::/0`
|
||||
must stay themselves, or a routing table becomes unreadable for no privacy gain.
|
||||
|
||||
**Still outstanding:** `measurement.schema.json` itself. Worth noting what this episode implies for
|
||||
it — much of a document's payload lives in `evidence`/`metrics`/`params`, which are per-test-type
|
||||
`JsonObject` by design and therefore *outside* any schema. A schema-driven anonymizer would have
|
||||
less coverage there than the name-plus-shape one now does, so the schema should be built for
|
||||
validation and external tooling, not as a replacement for the classifier.
|
||||
|
||||
### ULA prefixes are pseudonymized whole (2026-08-01)
|
||||
Spotted in a real uploaded run from the phone: the server had
|
||||
`fda1:3fb1:ff92:6696::2662` for a DNS server. The general IPv6 path preserves the leading two
|
||||
groups (deliberately — for a global address that keeps the ISP allocation, which is the
|
||||
diagnostically useful part), and for a ULA that passed through **32 of the 40 random bits** of the
|
||||
global ID.
|
||||
|
||||
ULA looks like the v6 equivalent of RFC1918 and the instinct is to treat it the same. That
|
||||
reasoning does not carry over, and the difference is the whole point: an RFC1918 prefix is shared
|
||||
by millions of networks and identifies none of them, while a ULA global ID is random and unique to
|
||||
one network by construction (RFC 4193). The prefix *is* the identifier — it is a network
|
||||
fingerprint that was surviving redaction.
|
||||
|
||||
Now pseudonymized as a unit, so two addresses on the same ULA subnet still land on the same
|
||||
pseudonymous prefix: "these hosts are on one network" survives, "this is *that* network" does not.
|
||||
Three tests, one of which uses the exact value observed on the wire.
|
||||
|
||||
Worth recording as a reasoning trap: I had originally raised this as "ULA should probably be kept
|
||||
verbatim, like RFC1918, for consistency". The surface analogy pointed the wrong way, and the
|
||||
correct answer was the opposite.
|
||||
|
||||
### Registry adopted everywhere; v6 findings renamed; Back works (2026-08-01)
|
||||
The findings registry was only adopted in `core-engine`. The app module still emitted seven codes
|
||||
as raw strings, so the registry test passed while codes existed outside it — including
|
||||
`ipv6.broken`, which fired on a real network and was in no registry at all.
|
||||
|
||||
All seven now reference registry entries for code, category and severity, so those three cannot
|
||||
disagree at a call site. A grep for `code = "…"` across the app, engine and probe modules returns
|
||||
nothing.
|
||||
|
||||
**`ipv6.*` → `v6.*`.** The third instance of rule 1: they declared `Category.IPV6` while the prefix
|
||||
map only knows `v6`, so `TestType.category("ipv6.broken")` fell through to *connectivity* and the
|
||||
finding rolled up under the wrong verdict light. The test-type registry already used `v6.`.
|
||||
|
||||
Two severities reconciled while merging:
|
||||
- `connectivity.captive_portal` is **medium**, not high. The registry had guessed high; the probe
|
||||
that emits it had always said medium, and the probe was the considered value — a captive portal
|
||||
on hotel wifi is what should be there, and logging in clears it. `connectivity.no_internet` is
|
||||
the high one, because nothing the user does locally fixes that.
|
||||
- `v6.not_offered` is **info, and the registry says it must stay info**. Most networks still do not
|
||||
offer IPv6 and that is not a fault; a warning here lights a yellow verdict on a healthy network,
|
||||
which teaches people to ignore the light.
|
||||
|
||||
Also: a `BackHandler` now returns from Settings/History to the run screen. The screen was a plain
|
||||
state variable with nothing connecting it to the back stack, so the system Back gesture left the
|
||||
app entirely. Enabled only when there is somewhere to go back to, so Back still exits from the run
|
||||
screen.
|
||||
|
||||
### Upstream throughput (server-v0.6.3, 2026-08-01)
|
||||
The mirror of the downstream case: the client generates the traffic and the server counts it. No
|
||||
grant is involved — the client is sending its own packets, so there is nothing to amplify — but it
|
||||
does need the server's tally, because **only the far end knows how much arrived**. Without that
|
||||
number a sender measures how fast it can *transmit*, which is usually just the speed of the local
|
||||
NIC and is a different question from the one being asked.
|
||||
|
||||
`TYPE_THROUGHPUT_UP` (0x0F) is counted and deliberately **never answered**: a reply would double
|
||||
the traffic and drag the return path into a measurement that is specifically about the outbound
|
||||
one.
|
||||
|
||||
The tally is a counter, not a list, and short-circuits **before** the observation log. A
|
||||
five-second run at 20 Mbps is around ten thousand packets; one struct each would turn a
|
||||
measurement into an allocation storm on a shared server, and nothing needs the per-packet detail
|
||||
since the client holds the send-side record. The gap between the two counts is the loss.
|
||||
|
||||
`direction=up` on the throughput action sends nothing — it zeroes the counter, so a second run in
|
||||
one session measures itself rather than inheriting the first one's packets. The live test asserts
|
||||
`received <= sent`, which is what catches a counter that was never reset.
|
||||
|
||||
Live against fmr: **3125 sent, 3125 counted, 0 % loss, 10.0 Mbit/s** at a 10 Mbit/s request, with
|
||||
`measures_network: false` — correct, since what arrived matched what was offered, so the path was
|
||||
never the constraint.
|
||||
|
||||
### Raw shell dumps leaked the whole LAN (2026-08-01)
|
||||
Found by running the Shizuku shell tier for the first time. The tier works — `tiers.shizuku: true`,
|
||||
`exec_path: UserService` (so the UserService binds on the OnePlus, as recorded), `runs_as
|
||||
shell(2000)`, 7/7 commands — and the run promptly uploaded **every MAC address on the local
|
||||
network** to fmr at the `balanced` level: router, phones, whatever else was on the wifi. Fourteen
|
||||
of them.
|
||||
|
||||
The probes embed raw command output verbatim (`ip neigh`, `ip route`, `id`), which is genuinely
|
||||
good evidence and also a complete household device inventory. The anonymizer could not see it:
|
||||
classification is by field name and by whole-value shape, and `ip_neigh` is one long string that is
|
||||
itself neither a MAC nor an address. measurement-schema.md §9 item 2 had flagged raw dumps as "hard
|
||||
to anonymize" and proposed dropping them from exports; nothing enforced either.
|
||||
|
||||
**Scrubbing beats dropping.** Identifiers inside any unclassified string are now replaced in place,
|
||||
using the same pseudonyms as everywhere else — so a MAC that appears both in a parsed field and in
|
||||
a raw dump still reads as one device. The dump stays readable and auditable: you can still see the
|
||||
neighbour table's shape, the host count, RFC1918 addresses and vendor prefixes. Dropping the
|
||||
evidence would have protected the same data while destroying the reason for collecting it.
|
||||
|
||||
Two implementation notes worth keeping:
|
||||
- **One pass, not three.** Sequential passes re-process their own output: once a MAC became
|
||||
`78:9a:18:xx:yy:zz`, the IPv6 pattern matched it — six hex groups separated by colons *is* an
|
||||
address — and destroyed the vendor prefix the MAC rule had just preserved. Ordered alternation
|
||||
resolves each position once, MAC first.
|
||||
- The patterns are conservative on purpose. A missed address gets caught by another rule or not at
|
||||
all; an over-eager one mangles timestamps and version strings, corrupting evidence to protect
|
||||
nothing.
|
||||
|
||||
`RealDocumentTest` runs the anonymizer over a captured run when `ECHOLOT_REAL_RUN` points at one,
|
||||
and fails on any MAC that survives. It self-skips otherwise, so no one's network is committed to the
|
||||
repo. Against the actual leaked document: **14 MACs in, 0 surviving.**
|
||||
|
||||
Also fixed: the Settings *Preview what an upload would send* button did nothing. It read
|
||||
`UiState.history`, which is empty until the History screen has been opened — the same root cause as
|
||||
the "0 run(s)" count. It now reads the archive directly, and says so when there is nothing to
|
||||
preview rather than silently ignoring the tap.
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
<!--
|
||||
SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
SPDX-License-Identifier: CC-BY-4.0
|
||||
-->
|
||||
|
||||
# Echolot findings registry
|
||||
|
||||
Closes open item 1 of `measurement-schema.md` §9.
|
||||
|
||||
A **finding code** is the stable, machine-readable half of a result. The prose around it changes
|
||||
freely; the code is what a dashboard groups by, what a diff between two runs keys on, and what
|
||||
someone greps a year of archived runs for. That only works if a code means exactly one thing,
|
||||
forever.
|
||||
|
||||
This document is the contract. It is kept in step with
|
||||
`echolot-app/core-measurement/.../FindingRegistry.kt` by a test that fails when either side has a
|
||||
code the other does not — a registry that drifts from its documentation is worse than none,
|
||||
because it looks authoritative.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **The prefix determines the category**, and the category determines which verdict light the
|
||||
finding rolls up into (§7.3). A `nat.*` code appearing under *connectivity* is not a naming
|
||||
quibble; it changes which light turns red. Two codes were renamed from `nat.*` to
|
||||
`connectivity.*` for exactly this reason.
|
||||
2. **One code per concept.** Two emitters independently produced `connectivity.downstream_loss`
|
||||
and `connectivity.loss_downstream` for the same claim before this registry existed. Anyone
|
||||
aggregating either would have silently seen half their data.
|
||||
3. **Codes are declared, not typed.** Emitters reference a `FindingSpec`, so a typo is a compile
|
||||
error and no two call sites can disagree about a finding's category or default severity.
|
||||
4. **Severity in the registry is the default.** An emitter may escalate for a specific run; it may
|
||||
not quietly reclassify the finding in general.
|
||||
5. **Say what is ruled out**, where that is the useful half. "Loss upstream" is worth far more
|
||||
when it also states that the return path is clean, because that halves where to look next.
|
||||
6. **Renaming a code is a breaking change** once runs are archived at scale. Before 1.0 it is
|
||||
cheap; after, it needs an alias and a deprecation window.
|
||||
|
||||
## Registry
|
||||
|
||||
### connectivity
|
||||
|
||||
| code | severity | means | rules out |
|
||||
|---|---|---|---|
|
||||
| `connectivity.udp_unreachable` | high | No UDP echo replies came back from the server at all. | — |
|
||||
| `connectivity.udp_unreachable_upstream` | high | The server received none of the probes, so traffic is dropped on the way out. | The return path: nothing arrived to be replied to. |
|
||||
| `connectivity.udp_loss` | medium | A large fraction of round-trip probes were lost, direction unknown. | — |
|
||||
| `connectivity.loss_upstream` | medium | Probes were lost on the way to the server. | The return path: replies came back for everything that arrived. |
|
||||
| `connectivity.loss_downstream` | medium | Packets were lost on the way back from the server. | The outbound path: the server received what it was answering. |
|
||||
| `connectivity.downstream_blocked` | high | Server-initiated packets never arrive, although round trips work. | Basic reachability: the path forwards replies, just not unsolicited traffic. |
|
||||
| `connectivity.downstream_reorder` | low | Downstream packets arrive in a different order than they were sent. | — |
|
||||
| `connectivity.captive_portal` | medium | A captive portal is intercepting connectivity checks. | — |
|
||||
| `connectivity.no_internet` | high | Android's own connectivity checks fail on this network. | — |
|
||||
|
||||
### mtu
|
||||
|
||||
| code | severity | means | rules out |
|
||||
|---|---|---|---|
|
||||
| `mtu.reduced_downstream` | low | The downstream path MTU is below the usual 1500 bytes. | — |
|
||||
| `mtu.downstream_blackhole` | medium | Datagrams above the path MTU are dropped downstream, fragmented or not. | — |
|
||||
| `mtu.fragments_blocked` | medium | IP fragments do not reach this device even when sent in order. | — |
|
||||
| `mtu.fragment_reorder_sensitive` | low | Fragments are delivered in order but dropped when reordered or delayed. | Fragmentation itself: in-order fragments arrive fine. |
|
||||
|
||||
### nat
|
||||
|
||||
| code | severity | means | rules out |
|
||||
|---|---|---|---|
|
||||
| `nat.udp_rebinding` | medium | A NAT remapped the UDP source port mid-flow. | — |
|
||||
| `nat.symmetric` | medium | The NAT assigns a different external port per destination. | — |
|
||||
|
||||
### perf
|
||||
|
||||
| code | severity | means | rules out |
|
||||
|---|---|---|---|
|
||||
| `perf.throughput_no_delivery` | high | No throughput traffic arrived, although the server sent it. | — |
|
||||
| `perf.throughput_below_offered` | low | Less throughput arrived than the server sent for the whole run. | — |
|
||||
|
||||
### dns
|
||||
|
||||
| code | severity | means | rules out |
|
||||
|---|---|---|---|
|
||||
| `dns.answer_rewritten` | high | A resolver returned an answer that differs from the authoritative record. | — |
|
||||
| `dns.authoritative_unreachable` | medium | The canary zone's authoritative server could not be reached. | — |
|
||||
|
||||
### v6
|
||||
|
||||
The prefix is `v6.`, matching the test-type registry (`v6.brokenness`, `v6.happy_eyeballs`, …).
|
||||
These were `ipv6.*` while declaring `Category.IPV6`; since the prefix map only knows `v6`, they
|
||||
rolled up under *connectivity* instead — the third occurrence of rule 1 being broken.
|
||||
|
||||
| code | severity | means | rules out |
|
||||
|---|---|---|---|
|
||||
| `v6.broken` | medium | IPv6 is configured on this network but does not work. | Absence of IPv6: it is provisioned, it simply fails. |
|
||||
| `v6.not_offered` | info | This network does not offer IPv6. | — |
|
||||
|
||||
`v6.not_offered` is **info and must stay info**. Most networks still do not offer IPv6 and that is
|
||||
not a fault; reporting it as a warning lights a yellow verdict on a healthy network, which teaches
|
||||
people to ignore the light — the one thing a diagnostic must never do.
|
||||
|
||||
## Adding a finding
|
||||
|
||||
1. Add a `FindingSpec` to `FindingRegistry`, and to its `all` list.
|
||||
2. Add the row here, under the section its prefix names.
|
||||
3. Emit it with `finding(FindingRegistry.YOUR_CODE, …)`.
|
||||
|
||||
The registry test checks 1 and 2 agree, that every prefix maps to the category it claims, and that
|
||||
no two entries share a code.
|
||||
@@ -269,7 +269,7 @@ The JSON Schema (machine-readable companion, `measurement.schema.json`, generate
|
||||
|
||||
| type | example fields | v2 anonymizer transform |
|
||||
|---|---|---|
|
||||
| `ip4`, `ip6` | addresses, routes, hops, DNS answers | prefix-preserving pseudonymization, consistent per document; well-known/reserved ranges kept verbatim |
|
||||
| `ip4`, `ip6` | addresses, routes, hops, DNS answers | prefix-preserving pseudonymization, consistent per document; well-known/reserved ranges kept verbatim. **Exception: ULA (`fc00::/7`) has its whole prefix pseudonymized as a unit.** It resembles RFC1918 but is not analogous: a ULA global ID is 40 random bits, unique to one network by construction (RFC 4193), so the prefix *is* the identifier, whereas `192.168.0.0/16` is shared by millions of networks and identifies none. Pseudonymizing it as a unit keeps "these hosts are on one subnet" while dropping "this is that subnet". |
|
||||
| `mac`, `bssid` | wifi, arp_watch | OUI kept, NIC part pseudonymized |
|
||||
| `fqdn` | DNS names, reverse lookups | per-label pseudonyms, public-suffix kept |
|
||||
| `ssid` | wifi | pseudonym |
|
||||
@@ -279,7 +279,8 @@ Free-text fields (`notes`, `error.detail`, dump excerpts from Shizuku parsers) c
|
||||
|
||||
## 9. Open items
|
||||
|
||||
1. Findings registry document — start alongside the first implemented tests.
|
||||
1. ~~Findings registry document~~ — done: `findings-registry.md`, kept in step with
|
||||
`FindingRegistry.kt` by a test that fails when the two disagree.
|
||||
2. Whether Shizuku raw-dump excerpts (dumpsys/ip output) are embedded in `evidence` verbatim (auditable, but large and hard to anonymize) or parsed-only with an optional "attach raw dumps" toggle. Proposal: toggle, default on for local archive, default off for export.
|
||||
3. Peer-mode documents: each device produces its own run; the coordinator embeds the peer's findings summary and cross-references by `run.id`. Full merge format deferred.
|
||||
4. Size guardrails: soft cap 20 MB uncompressed per run; trains beyond that downsample evidence (keep aggregates + first/last N + all anomalies) and record `"evidence_truncated": true`.
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
package app.echo_lot.app
|
||||
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.safeDrawingPadding
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
@@ -39,7 +40,7 @@ fun HistoryScreen(
|
||||
onDelete: (String) -> Unit,
|
||||
onBack: () -> Unit,
|
||||
) {
|
||||
Column(Modifier.fillMaxWidth().padding(16.dp), verticalArrangement = Arrangement.spacedBy(10.dp)) {
|
||||
Column(Modifier.fillMaxWidth().safeDrawingPadding().padding(16.dp), verticalArrangement = Arrangement.spacedBy(10.dp)) {
|
||||
Row(verticalAlignment = Alignment.CenterVertically) {
|
||||
TextButton(onClick = onBack) { Text("‹ Back") }
|
||||
Text("History", style = MaterialTheme.typography.titleLarge)
|
||||
@@ -71,12 +72,20 @@ fun HistoryScreen(
|
||||
)
|
||||
}
|
||||
Text(
|
||||
"${r.findingCount} finding(s) · ${r.sizeBytes / 1024} kB · ${r.anonymization}",
|
||||
"${r.findingCount} finding(s) · ${r.sizeBytes / 1024} kB · " +
|
||||
"kept complete on this device",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
)
|
||||
// The upload line names the level the upload was made at, not the
|
||||
// archive's. They describe different documents, and showing the archive's
|
||||
// level here claimed more had left the device than actually did.
|
||||
Text(
|
||||
if (r.uploaded) "uploaded to ${r.uploadedTo ?: "a server"}"
|
||||
else "on this device only",
|
||||
if (r.uploaded) {
|
||||
"uploaded to ${r.uploadedTo ?: "a server"}" +
|
||||
(r.uploadedAs?.let { " as $it" } ?: "")
|
||||
} else {
|
||||
"on this device only"
|
||||
},
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = if (r.uploaded) Color(0xFF7FD17F) else Color(0xFFBBBBBB),
|
||||
)
|
||||
|
||||
@@ -85,19 +85,26 @@ class MainActivity : ComponentActivity() {
|
||||
finish()
|
||||
}
|
||||
}
|
||||
// Without this, the system Back gesture leaves the activity from Settings or
|
||||
// History instead of returning to the run screen — the screen is a plain state
|
||||
// variable, so nothing connects it to the back stack. Registered only when
|
||||
// there is somewhere to go back to, so Back still exits from the run screen.
|
||||
androidx.activity.compose.BackHandler(enabled = screen != Screen.RUN) {
|
||||
screen = Screen.RUN
|
||||
}
|
||||
|
||||
when (screen) {
|
||||
Screen.SETTINGS -> SettingsScreen(
|
||||
settings = vm.settings,
|
||||
archivedRuns = vm.state.history.size,
|
||||
archivedRuns = vm.archivedRunCount(),
|
||||
archivedBytes = vm.archivedBytes(),
|
||||
onApplyRetention = vm::applyRetention,
|
||||
onDeleteAll = vm::deleteAllRuns,
|
||||
onPreviewUpload = {
|
||||
// Preview the newest run, since that is the one the user just made
|
||||
// and the one they are deciding about.
|
||||
vm.state.history.firstOrNull()?.let { r ->
|
||||
lifecycleScope.launch { preview = vm.uploadPreview(r.id) }
|
||||
}
|
||||
// Straight from the archive: the newest run is the one the user just
|
||||
// made and the one they are deciding about. Always shows something,
|
||||
// even when there is nothing to preview yet.
|
||||
lifecycleScope.launch { preview = vm.previewNewestRun() }
|
||||
},
|
||||
onCheckServer = vm::checkServer,
|
||||
onEnroll = vm::enroll,
|
||||
|
||||
@@ -164,8 +164,10 @@ class RunStore(context: Context, private val settings: Settings) {
|
||||
)
|
||||
val body = redactedForUpload(docJson, level)
|
||||
val reply = client.uploadRun(settings.serverCredential, body)
|
||||
archive.markUploaded(runId, profile.name)
|
||||
UploadOutcome.Sent(profile.name, "as $level, ${body.toByteArray().size} bytes: ${reply.take(120)}")
|
||||
archive.markUploaded(runId, profile.name, level.wire)
|
||||
// Deliberately not echoing `reply`: it is the server's index entry as raw JSON, and
|
||||
// it ended up rendered verbatim in the UI. Size and level are what a person wants.
|
||||
UploadOutcome.Sent(profile.name, "as $level, ${body.toByteArray().size} bytes")
|
||||
} catch (e: VersionRefused) {
|
||||
UploadOutcome.Incompatible(e.message ?: "the server refused this app's version")
|
||||
} catch (e: UploadRefused) {
|
||||
|
||||
@@ -194,8 +194,27 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
store.read(id)?.let { store.redactedForUpload(it) }
|
||||
}
|
||||
|
||||
/**
|
||||
* Preview of the most recent run, read from the archive rather than from [UiState.history].
|
||||
*
|
||||
* The history list is only populated once the History screen has been opened, so a preview
|
||||
* driven from it did nothing at all on a freshly-opened Settings screen — a button that
|
||||
* silently does nothing is worse than one that says why.
|
||||
*/
|
||||
suspend fun previewNewestRun(): String = withContext(Dispatchers.IO) {
|
||||
val newest = store.list().firstOrNull()
|
||||
?: return@withContext "No archived runs yet. Run a measurement first, then this will " +
|
||||
"show exactly what an upload would send."
|
||||
store.read(newest.id)?.let { store.redactedForUpload(it) }
|
||||
?: "That run could not be read back from the archive."
|
||||
}
|
||||
|
||||
fun archivedBytes(): Long = store.totalBytes()
|
||||
|
||||
/** Counted from the archive itself, not from [UiState.history], which is empty until the
|
||||
* history screen has been opened - the two disagreeing read as data loss. */
|
||||
fun archivedRunCount(): Int = store.list().size
|
||||
|
||||
/** Redeems an enrollment link, from a paste or from an echolot:// deep link. */
|
||||
fun enroll(link: String, deviceName: String? = android.os.Build.MODEL) {
|
||||
viewModelScope.launch {
|
||||
@@ -352,8 +371,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
when {
|
||||
ev.contains("\"captive_portal\"") -> out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "connectivity.captive_portal", category = Category.CONNECTIVITY,
|
||||
severity = Severity.MEDIUM, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = FindingRegistry.CAPTIVE_PORTAL.code,
|
||||
category = FindingRegistry.CAPTIVE_PORTAL.category,
|
||||
severity = FindingRegistry.CAPTIVE_PORTAL.severity, confidence = Confidence.HIGH,
|
||||
title = "Captive portal intercepting connections",
|
||||
description = "The generate_204 check returned a redirect or a page instead of HTTP 204 — a captive portal (login/splash page) is intercepting traffic on this network.",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
@@ -361,8 +381,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
)
|
||||
t.status == TestStatus.FAILED -> out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "connectivity.no_internet", category = Category.CONNECTIVITY,
|
||||
severity = Severity.HIGH, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = FindingRegistry.NO_INTERNET.code,
|
||||
category = FindingRegistry.NO_INTERNET.category,
|
||||
severity = FindingRegistry.NO_INTERNET.severity, confidence = Confidence.HIGH,
|
||||
title = "No working internet on any network",
|
||||
description = "Android's own generate_204 connectivity checks failed on every active network (no HTTP 204) — this device has no validated internet path.",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
@@ -375,8 +396,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
if (ev.contains("MISMATCH")) {
|
||||
out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "dns.answer_rewritten", category = Category.DNS,
|
||||
severity = Severity.HIGH, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = FindingRegistry.DNS_ANSWER_REWRITTEN.code,
|
||||
category = FindingRegistry.DNS_ANSWER_REWRITTEN.category,
|
||||
severity = FindingRegistry.DNS_ANSWER_REWRITTEN.severity, confidence = Confidence.HIGH,
|
||||
title = "DNS answers are being rewritten",
|
||||
description = "A canary reference record returned different RDATA than the spec-defined ground truth — something on the path is rewriting DNS answers (interception, filtering, or a middlebox).",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
@@ -385,8 +407,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
} else if (ev.contains("\"reached_authoritative\":false")) {
|
||||
out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "dns.authoritative_unreachable", category = Category.DNS,
|
||||
severity = Severity.MEDIUM, confidence = Confidence.MEDIUM,
|
||||
id = ids.uuid(), code = FindingRegistry.DNS_AUTHORITATIVE_UNREACHABLE.code,
|
||||
category = FindingRegistry.DNS_AUTHORITATIVE_UNREACHABLE.category,
|
||||
severity = FindingRegistry.DNS_AUTHORITATIVE_UNREACHABLE.severity, confidence = Confidence.MEDIUM,
|
||||
title = "Canary queries don't reach the authoritative server",
|
||||
description = "A per-run nonce name (which cannot be cached) was not answered by the canary server — the resolver is intercepting or failing to reach it.",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
@@ -399,8 +422,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
if (ev.contains("address/port-dependent (symmetric NAT")) {
|
||||
out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "nat.symmetric", category = Category.NAT,
|
||||
severity = Severity.MEDIUM, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = FindingRegistry.NAT_SYMMETRIC.code,
|
||||
category = FindingRegistry.NAT_SYMMETRIC.category,
|
||||
severity = FindingRegistry.NAT_SYMMETRIC.severity, confidence = Confidence.HIGH,
|
||||
title = "Symmetric NAT — peer-to-peer connections need a relay",
|
||||
description = "The NAT assigns a different external port per destination (address/port-dependent mapping). Direct peer-to-peer connections (calls, games, file transfer) will usually fail and fall back to relays.",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
@@ -417,8 +441,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
if (ipv6Provisioned(networks)) {
|
||||
out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "ipv6.broken", category = Category.IPV6,
|
||||
severity = Severity.MEDIUM, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = FindingRegistry.V6_BROKEN.code,
|
||||
category = FindingRegistry.V6_BROKEN.category,
|
||||
severity = FindingRegistry.V6_BROKEN.severity, confidence = Confidence.HIGH,
|
||||
title = "IPv6 is configured but not working",
|
||||
description = "This network advertises IPv6 (a global address and/or a default route), but ICMPv6 got no reply on any network. Half-configured IPv6 is worse than none: connections try IPv6 first and stall before falling back.",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
@@ -427,8 +452,9 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
||||
} else {
|
||||
out.add(
|
||||
Finding(
|
||||
id = ids.uuid(), code = "ipv6.not_offered", category = Category.IPV6,
|
||||
severity = Severity.INFO, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = FindingRegistry.V6_NOT_OFFERED.code,
|
||||
category = FindingRegistry.V6_NOT_OFFERED.category,
|
||||
severity = FindingRegistry.V6_NOT_OFFERED.severity, confidence = Confidence.HIGH,
|
||||
title = "IPv4-only network (no IPv6 offered)",
|
||||
description = "No IPv6 address or default route was provisioned, so IPv6 tests could not run. This is normal — many networks are still IPv4-only and it is not a fault.",
|
||||
evidenceRefs = listOf(EvidenceRef(t.id)),
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
package app.echo_lot.app
|
||||
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.safeDrawingPadding
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
@@ -15,6 +16,7 @@ import androidx.compose.foundation.verticalScroll
|
||||
import androidx.compose.material3.Button
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.FilterChip
|
||||
import androidx.compose.material3.LocalContentColor
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.OutlinedTextField
|
||||
import androidx.compose.material3.Switch
|
||||
@@ -66,7 +68,7 @@ fun SettingsScreen(
|
||||
var serverCred by remember { mutableStateOf(settings.serverCredential) }
|
||||
|
||||
Column(
|
||||
Modifier.fillMaxWidth().verticalScroll(rememberScrollState()).padding(16.dp),
|
||||
Modifier.fillMaxWidth().safeDrawingPadding().verticalScroll(rememberScrollState()).padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Row(verticalAlignment = Alignment.CenterVertically) {
|
||||
@@ -129,12 +131,21 @@ fun SettingsScreen(
|
||||
}
|
||||
Text(privacyExplanation(privacy), style = MaterialTheme.typography.bodySmall)
|
||||
|
||||
// At FULL nothing is pseudonymized, so a salt has nothing to act on. Shown
|
||||
// disabled rather than hidden: the setting is still stored and still applies the
|
||||
// moment the level changes, and a control that vanishes hides that fact.
|
||||
Toggle(
|
||||
label = "Stable pseudonyms across runs",
|
||||
detail = "Lets you compare uploaded runs over time (same SSID reads the same " +
|
||||
"each time). It also links your uploads together, so leave it off on a " +
|
||||
"server you don't run yourself.",
|
||||
checked = stableSalt,
|
||||
detail = if (privacy == PrivacyLevel.FULL) {
|
||||
"Not used at this level — nothing is pseudonymized, so there is nothing " +
|
||||
"to keep stable. Choose balanced or strict to use this."
|
||||
} else {
|
||||
"Lets you compare uploaded runs over time (same SSID reads the same " +
|
||||
"each time). It also links your uploads together, so leave it off on " +
|
||||
"a server you don't run yourself."
|
||||
},
|
||||
checked = stableSalt && privacy != PrivacyLevel.FULL,
|
||||
enabled = privacy != PrivacyLevel.FULL,
|
||||
) { stableSalt = it; settings.stableSalt = it }
|
||||
|
||||
TextButton(onClick = onPreviewUpload) { Text("Preview what an upload would send") }
|
||||
@@ -236,13 +247,22 @@ private fun privacyExplanation(level: PrivacyLevel): String = when (level) {
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun Toggle(label: String, detail: String, checked: Boolean, onChange: (Boolean) -> Unit) {
|
||||
private fun Toggle(
|
||||
label: String,
|
||||
detail: String,
|
||||
checked: Boolean,
|
||||
enabled: Boolean = true,
|
||||
onChange: (Boolean) -> Unit,
|
||||
) {
|
||||
Row(Modifier.fillMaxWidth(), verticalAlignment = Alignment.Top) {
|
||||
Column(Modifier.weight(1f)) {
|
||||
Text(label, style = MaterialTheme.typography.bodyMedium)
|
||||
Text(detail, style = MaterialTheme.typography.bodySmall)
|
||||
// Dimmed together with the switch, so "this does nothing right now" reads at a glance
|
||||
// instead of only on close inspection.
|
||||
val alpha = if (enabled) 1f else 0.5f
|
||||
Text(label, style = MaterialTheme.typography.bodyMedium, color = LocalContentColor.current.copy(alpha = alpha))
|
||||
Text(detail, style = MaterialTheme.typography.bodySmall, color = LocalContentColor.current.copy(alpha = alpha))
|
||||
}
|
||||
Switch(checked = checked, onCheckedChange = onChange)
|
||||
Switch(checked = checked, onCheckedChange = onChange, enabled = enabled)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -31,10 +31,23 @@ data class ArchivedRun(
|
||||
val verdict: String? = null,
|
||||
@SerialName("finding_count") val findingCount: Int = 0,
|
||||
@SerialName("size_bytes") val sizeBytes: Long = 0,
|
||||
/**
|
||||
* How the *archived* document is redacted. Always "full" in practice, because the archive
|
||||
* deliberately keeps the unredacted run - see the package doc. This is not what was uploaded.
|
||||
*/
|
||||
val anonymization: String = "full",
|
||||
/** Whether this run has been accepted by a server, so history can show what is backed up. */
|
||||
val uploaded: Boolean = false,
|
||||
@SerialName("uploaded_to") val uploadedTo: String? = null,
|
||||
/**
|
||||
* The level the run was *uploaded* at, which is a different document from the archived one.
|
||||
*
|
||||
* Kept separately because conflating the two is actively misleading: the history row showed
|
||||
* the archive's own level ("full") directly beneath "uploaded to fmr", which reads as "the
|
||||
* complete data was uploaded" when a redacted copy had been sent. A privacy display that
|
||||
* overstates what left the device is worse than none.
|
||||
*/
|
||||
@SerialName("uploaded_as") val uploadedAs: String? = null,
|
||||
)
|
||||
|
||||
/**
|
||||
@@ -119,7 +132,7 @@ class RunArchive(private val dir: File, private val now: () -> Long = System::cu
|
||||
fun deleteAll(): Int = list().count { delete(it.id) }
|
||||
|
||||
/** Records that a server accepted this run, so history can distinguish backed-up from local. */
|
||||
fun markUploaded(id: String, serverName: String) {
|
||||
fun markUploaded(id: String, serverName: String, uploadedAs: String? = null) {
|
||||
val f = File(dir, safe(id) + META_EXT)
|
||||
val meta = runCatching { json.decodeFromString(ArchivedRun.serializer(), f.readText()) }.getOrNull()
|
||||
?: return
|
||||
@@ -127,7 +140,7 @@ class RunArchive(private val dir: File, private val now: () -> Long = System::cu
|
||||
f,
|
||||
json.encodeToString(
|
||||
ArchivedRun.serializer(),
|
||||
meta.copy(uploaded = true, uploadedTo = serverName),
|
||||
meta.copy(uploaded = true, uploadedTo = serverName, uploadedAs = uploadedAs),
|
||||
),
|
||||
)
|
||||
}
|
||||
@@ -181,7 +194,10 @@ class RunArchive(private val dir: File, private val now: () -> Long = System::cu
|
||||
id = id,
|
||||
savedAtEpochMs = now(),
|
||||
startedAt = run["started_at"]?.jsonPrimitive?.content,
|
||||
verdict = doc["summary"]?.jsonObject?.get("verdict")?.jsonPrimitive?.content,
|
||||
// The schema calls it `overall` (Summary.overall); reading `verdict` here silently
|
||||
// yielded null for every run, so the history list's most prominent element - the
|
||||
// coloured verdict - was blank on every row.
|
||||
verdict = doc["summary"]?.jsonObject?.get("overall")?.jsonPrimitive?.content,
|
||||
findingCount = (doc["findings"] as? kotlinx.serialization.json.JsonArray)?.size ?: 0,
|
||||
sizeBytes = size,
|
||||
anonymization = run["privacy"]?.jsonObject?.get("anonymization")?.jsonPrimitive?.content ?: "full",
|
||||
|
||||
@@ -25,7 +25,7 @@ class RunArchiveTest {
|
||||
private fun doc(id: String, findings: Int = 1, pad: Int = 0): String {
|
||||
val f = (1..findings).joinToString(",") { """{"id":"f$it"}""" }
|
||||
return """{"run":{"id":"$id","started_at":"2026-08-01T10:00:00Z","privacy":{"anonymization":"balanced"}},""" +
|
||||
""""findings":[$f],"summary":{"verdict":"warn"},"pad":"${"x".repeat(pad)}"}"""
|
||||
""""findings":[$f],"summary":{"overall":"warn"},"pad":"${"x".repeat(pad)}"}"""
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -120,13 +120,40 @@ class RunArchiveTest {
|
||||
fun uploadStateIsRecorded() {
|
||||
val a = archive()
|
||||
a.save(doc("run-1"))
|
||||
a.markUploaded("run-1", "fmr")
|
||||
a.markUploaded("run-1", "fmr", "balanced")
|
||||
val meta = a.list().single()
|
||||
assertTrue(meta.uploaded)
|
||||
assertEquals("fmr", meta.uploadedTo)
|
||||
assertEquals("run-1", meta.id, "marking upload must not disturb the rest of the entry")
|
||||
}
|
||||
|
||||
// The archive's own level and the level a run was uploaded at describe *different documents*.
|
||||
// Showing the archive's ("full", because the archive is deliberately unredacted) next to
|
||||
// "uploaded to fmr" reads as "the complete data was uploaded" when a redacted copy was sent —
|
||||
// a privacy display that overstates what left the device is worse than none.
|
||||
@Test
|
||||
fun theUploadedLevelIsRecordedSeparatelyFromTheArchivedOne() {
|
||||
val a = archive()
|
||||
// A real archived document carries no privacy stamp: the anonymizer never runs on the
|
||||
// archive. The shared doc() fixture has one, which is exactly the unrealism that let this
|
||||
// confusion through in the first place.
|
||||
a.save("""{"run":{"id":"run-1"},"findings":[],"summary":{"overall":"green"}}""")
|
||||
a.markUploaded("run-1", "fmr", "balanced")
|
||||
val meta = a.list().single()
|
||||
assertEquals("full", meta.anonymization, "the archived copy is unredacted, by design")
|
||||
assertEquals("balanced", meta.uploadedAs, "the uploaded copy was redacted, and must say so")
|
||||
}
|
||||
|
||||
// The verdict is read from `summary.overall` — the schema's actual field name. Reading
|
||||
// `summary.verdict` silently yielded null for every run, so the history list's most prominent
|
||||
// element was blank on every row while everything else looked fine.
|
||||
@Test
|
||||
fun theVerdictComesFromTheSchemasOverallField() {
|
||||
val a = archive()
|
||||
a.save("""{"run":{"id":"r1"},"findings":[],"summary":{"overall":"yellow"}}""")
|
||||
assertEquals("yellow", a.list().single().verdict)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun deleteRemovesBothFiles() {
|
||||
val a = archive()
|
||||
|
||||
+17
-9
@@ -118,7 +118,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
if (!inOrder) {
|
||||
findings.add(
|
||||
finding(
|
||||
"mtu.fragments_blocked", Category.MTU, Severity.MEDIUM, testId,
|
||||
FindingRegistry.FRAGMENTS_BLOCKED, testId,
|
||||
"IP fragments do not reach this device",
|
||||
"A fragmented datagram sent in the normal order never arrived. Anything that " +
|
||||
"relies on fragmentation — large DNS answers over UDP, some VPN traffic — " +
|
||||
@@ -133,7 +133,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
}.joinToString(" or ")
|
||||
findings.add(
|
||||
finding(
|
||||
"mtu.fragment_reorder_sensitive", Category.MTU, Severity.LOW, testId,
|
||||
FindingRegistry.FRAGMENT_REORDER_SENSITIVE, testId,
|
||||
"Fragments are dropped when they arrive $which",
|
||||
"In-order fragments are delivered, but the same datagram sent $which is not. " +
|
||||
"Something on the path only reassembles when the first fragment (the one " +
|
||||
@@ -198,7 +198,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
if (ipMtu < 1500) {
|
||||
findings.add(
|
||||
finding(
|
||||
"mtu.reduced_downstream", Category.MTU, Severity.LOW, df.test.id,
|
||||
FindingRegistry.MTU_REDUCED_DOWNSTREAM, df.test.id,
|
||||
"Downstream path MTU is $ipMtu bytes, below 1500",
|
||||
"The largest datagram that reached this device without fragmenting was " +
|
||||
"$pathMtu bytes of payload ($ipMtu on the wire). Tunnels (PPPoE, VPN, " +
|
||||
@@ -213,7 +213,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
if (fragLargest <= pathMtu && sizes.any { it > pathMtu }) {
|
||||
findings.add(
|
||||
finding(
|
||||
"mtu.downstream_blackhole", Category.MTU, Severity.MEDIUM, frag.test.id,
|
||||
FindingRegistry.MTU_DOWNSTREAM_BLACKHOLE, frag.test.id,
|
||||
"Datagrams above $pathMtu bytes are dropped downstream, fragmented or not",
|
||||
"Nothing larger than $pathMtu bytes arrived, even when the network was " +
|
||||
"free to fragment it. Traffic that relies on large responses will " +
|
||||
@@ -226,7 +226,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
if (train.received == 0) {
|
||||
findings.add(
|
||||
finding(
|
||||
"connectivity.downstream_blocked", Category.CONNECTIVITY, Severity.HIGH, train.test.id,
|
||||
FindingRegistry.DOWNSTREAM_BLOCKED, train.test.id,
|
||||
"No server-initiated packets arrived",
|
||||
"The server sent ${train.sent} packets toward this device and none arrived, " +
|
||||
"while the round-trip echo worked. Something on the path forwards replies " +
|
||||
@@ -236,7 +236,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
} else if (train.lossPct >= 5.0) {
|
||||
findings.add(
|
||||
finding(
|
||||
"connectivity.downstream_loss", Category.CONNECTIVITY, Severity.MEDIUM, train.test.id,
|
||||
FindingRegistry.LOSS_DOWNSTREAM, train.test.id,
|
||||
"Downstream loss of ${round1(train.lossPct)}%",
|
||||
"${train.sent - train.received} of ${train.sent} packets sent toward this " +
|
||||
"device were lost. Downstream loss is invisible to a round-trip test, " +
|
||||
@@ -247,7 +247,7 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
if (train.reordered > 0) {
|
||||
findings.add(
|
||||
finding(
|
||||
"connectivity.downstream_reorder", Category.CONNECTIVITY, Severity.LOW, train.test.id,
|
||||
FindingRegistry.DOWNSTREAM_REORDER, train.test.id,
|
||||
"${train.reordered} downstream packet(s) arrived out of order",
|
||||
"Packets arrived in a different order than they were sent. Usually per-packet " +
|
||||
"load balancing across links; harmless for most traffic, not for all of it.",
|
||||
@@ -414,9 +414,17 @@ class DownstreamMeasurement(private val ids: IdSource) {
|
||||
|
||||
// ---- helpers ----------------------------------------------------------------------
|
||||
|
||||
private fun finding(code: String, cat: Category, sev: Severity, testId: String, title: String, desc: String) =
|
||||
/**
|
||||
* Builds a finding from a registry entry, which supplies the code, category and severity.
|
||||
*
|
||||
* Taking a [FindingSpec] rather than three loose values is the point: a typo becomes a
|
||||
* compile error, and two call sites cannot disagree about which category a finding belongs
|
||||
* to - a disagreement that would split one fault across two verdict lights.
|
||||
*/
|
||||
private fun finding(spec: FindingSpec, testId: String, title: String, desc: String) =
|
||||
Finding(
|
||||
id = ids.uuid(), code = code, category = cat, severity = sev, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = spec.code, category = spec.category, severity = spec.severity,
|
||||
confidence = Confidence.HIGH,
|
||||
title = title, description = desc, evidenceRefs = listOf(EvidenceRef(testId)),
|
||||
)
|
||||
|
||||
|
||||
@@ -47,6 +47,14 @@ class ServerMeasurement(
|
||||
* it is a flag rather than an assumption.
|
||||
*/
|
||||
val downstream: Boolean = true,
|
||||
/**
|
||||
* Throughput moves real data — a 5-second run at 50 Mbps is about 30 MB — so it is off
|
||||
* unless asked for. On a metered mobile connection that is the user's money, and a
|
||||
* measurement tool that spends it without being told to is not one people keep installed.
|
||||
*/
|
||||
val throughput: Boolean = false,
|
||||
@Suppress("unused") val throughputSeconds: Int = 5,
|
||||
@Suppress("unused") val throughputKbps: Int = 50_000,
|
||||
)
|
||||
|
||||
fun run(cfg: Config): MeasurementDocument {
|
||||
@@ -89,6 +97,15 @@ class ServerMeasurement(
|
||||
tests.addAll(dsTests)
|
||||
allFindings.addAll(dsFindings)
|
||||
}
|
||||
|
||||
if (cfg.throughput && profile.supports("throughput")) {
|
||||
val (tpTest, tpFindings) = ThroughputMeasurement(ids).run(
|
||||
cfg.credential, session.sessionId, control, ps, sessionRef = "sess-1",
|
||||
durationS = cfg.throughputSeconds, kbps = cfg.throughputKbps,
|
||||
)
|
||||
tests.add(tpTest)
|
||||
allFindings.addAll(tpFindings)
|
||||
}
|
||||
}
|
||||
|
||||
control.deleteSession(cfg.credential, session.sessionId)
|
||||
@@ -191,11 +208,11 @@ class ServerMeasurement(
|
||||
|
||||
val findings = ArrayList<Finding>()
|
||||
if (received == 0) {
|
||||
findings.add(finding("nat.udp_unreachable", Category.CONNECTIVITY, Severity.HIGH, testId,
|
||||
findings.add(finding(FindingRegistry.UDP_UNREACHABLE, testId,
|
||||
"No UDP echo replies from the server",
|
||||
"Every ECHO probe to the server's UDP data plane was lost — the path blocks or drops the session's UDP traffic."))
|
||||
} else if (lossPct >= 20.0) {
|
||||
findings.add(finding("connectivity.udp_loss", Category.CONNECTIVITY, Severity.MEDIUM, testId,
|
||||
findings.add(finding(FindingRegistry.UDP_LOSS, testId,
|
||||
"High UDP loss to the server (${round1(lossPct)}%)",
|
||||
"A large fraction of ECHO probes were lost, indicating an unreliable UDP path."))
|
||||
}
|
||||
@@ -203,14 +220,14 @@ class ServerMeasurement(
|
||||
directional?.let { d ->
|
||||
when {
|
||||
d.noneReachedServer && received == 0 -> findings.add(
|
||||
finding("nat.udp_unreachable_upstream", Category.CONNECTIVITY, Severity.HIGH, testId,
|
||||
finding(FindingRegistry.UDP_UNREACHABLE_UPSTREAM, testId,
|
||||
"Nothing reached the server",
|
||||
"The server received none of the ${d.sent} probes, so the traffic is being " +
|
||||
"dropped on the way out, not on the way back. A firewall or NAT on " +
|
||||
"this side of the path is the place to look."),
|
||||
)
|
||||
d.lossUpstreamPct >= 2.0 -> findings.add(
|
||||
finding("connectivity.loss_upstream", Category.CONNECTIVITY, Severity.MEDIUM, testId,
|
||||
finding(FindingRegistry.LOSS_UPSTREAM, testId,
|
||||
"${d.lossUpstreamPct} % of probes were lost on the way to the server",
|
||||
"${d.lostUpstream} of ${d.sent} probes never reached the server. The " +
|
||||
"return path is not implicated: replies came back for everything that " +
|
||||
@@ -219,7 +236,7 @@ class ServerMeasurement(
|
||||
}
|
||||
if (d.lossDownstreamPct >= 2.0) {
|
||||
findings.add(
|
||||
finding("connectivity.loss_downstream", Category.CONNECTIVITY, Severity.MEDIUM, testId,
|
||||
finding(FindingRegistry.LOSS_DOWNSTREAM, testId,
|
||||
"${d.lossDownstreamPct} % of replies were lost on the way back",
|
||||
"The server received ${d.seenByServer} probes and answered them, but " +
|
||||
"${d.lostDownstream} of those replies never arrived. The outbound path " +
|
||||
@@ -229,7 +246,7 @@ class ServerMeasurement(
|
||||
}
|
||||
|
||||
if (natRebinding) {
|
||||
findings.add(finding("nat.udp_rebinding", Category.NAT, Severity.MEDIUM, testId,
|
||||
findings.add(finding(FindingRegistry.NAT_UDP_REBINDING, testId,
|
||||
"NAT remapped the UDP source port mid-flow",
|
||||
"The server observed more than one source port for this session (${observedPorts.joinToString()}), i.e. a NAT with a short UDP mapping or per-packet remapping."))
|
||||
}
|
||||
@@ -259,9 +276,17 @@ class ServerMeasurement(
|
||||
}
|
||||
}
|
||||
|
||||
private fun finding(code: String, cat: Category, sev: Severity, testId: String, title: String, desc: String) =
|
||||
/**
|
||||
* Builds a finding from a registry entry, which supplies the code, category and severity.
|
||||
*
|
||||
* Taking a [FindingSpec] rather than three loose values is the point: a typo becomes a
|
||||
* compile error, and two call sites cannot disagree about which category a finding belongs
|
||||
* to - a disagreement that would split one fault across two verdict lights.
|
||||
*/
|
||||
private fun finding(spec: FindingSpec, testId: String, title: String, desc: String) =
|
||||
Finding(
|
||||
id = ids.uuid(), code = code, category = cat, severity = sev, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = spec.code, category = spec.category, severity = spec.severity,
|
||||
confidence = Confidence.HIGH,
|
||||
title = title, description = desc, evidenceRefs = listOf(EvidenceRef(testId)),
|
||||
)
|
||||
|
||||
|
||||
+141
-4
@@ -120,7 +120,7 @@ class ThroughputMeasurement(private val ids: IdSource) {
|
||||
sender == null -> Unit // no sender report: nothing can be concluded, so nothing is
|
||||
received.isEmpty() -> findings.add(
|
||||
finding(
|
||||
"perf.throughput_no_delivery", Category.PERFORMANCE, Severity.HIGH, testId,
|
||||
FindingRegistry.THROUGHPUT_NO_DELIVERY, testId,
|
||||
"No throughput traffic arrived",
|
||||
"The server sent ${sender.packets} packets and none arrived. This is a " +
|
||||
"connectivity fault rather than a slow link.",
|
||||
@@ -128,7 +128,7 @@ class ThroughputMeasurement(private val ids: IdSource) {
|
||||
)
|
||||
networkLimited -> findings.add(
|
||||
finding(
|
||||
"perf.throughput_below_offered", Category.PERFORMANCE, Severity.LOW, testId,
|
||||
FindingRegistry.THROUGHPUT_BELOW_OFFERED, testId,
|
||||
"Downstream throughput ${receivedKbps / 1000} Mbit/s, below the " +
|
||||
"${sender.kbps / 1000} Mbit/s offered",
|
||||
"The server sent at ${sender.kbps / 1000} Mbit/s for the full run and " +
|
||||
@@ -147,6 +147,117 @@ class ThroughputMeasurement(private val ids: IdSource) {
|
||||
) to findings
|
||||
}
|
||||
|
||||
/**
|
||||
* Upstream throughput: the client sends, the server counts.
|
||||
*
|
||||
* The mirror image of the downstream case, and it needs no grant — the client is generating
|
||||
* its own traffic, so there is no amplification to gate. What it does need is the server's
|
||||
* count: only the far end knows how much arrived, and without that number a sender can
|
||||
* measure how fast it can *transmit*, which is not the same question and is usually just the
|
||||
* speed of the local NIC.
|
||||
*/
|
||||
fun runUpstream(
|
||||
credential: String,
|
||||
sessionId: String,
|
||||
control: ControlClient,
|
||||
probe: ProbeSession,
|
||||
sessionRef: String,
|
||||
durationS: Int = 5,
|
||||
kbps: Int = 20_000,
|
||||
sizeBytes: Int = 1200,
|
||||
): Pair<Test, List<Finding>> {
|
||||
val testId = ids.uuid()
|
||||
val started = ids.monoNs()
|
||||
|
||||
// Zeroes the server's counter so this run measures itself rather than inheriting the
|
||||
// packets of an earlier one on the same session.
|
||||
val reply = runCatching {
|
||||
control.action(credential, sessionId, """{"action":"throughput","direction":"up"}""")
|
||||
}
|
||||
if (reply.isFailure) {
|
||||
return Test(
|
||||
id = testId, type = TestType.PERF_THROUGHPUT_UDP, sessionRef = sessionRef, tier = Tier.APP,
|
||||
startedMonoNs = started, endedMonoNs = ids.monoNs(),
|
||||
status = TestStatus.UNSUPPORTED,
|
||||
error = TestError("action_refused", reply.exceptionOrNull()?.message ?: "refused"),
|
||||
) to emptyList()
|
||||
}
|
||||
|
||||
val sent = probe.sendThroughput(durationS * 1000L, kbps, sizeBytes)
|
||||
// A moment for the tail of the run to arrive; counting still-in-flight packets as lost
|
||||
// would inflate the loss figure by whatever the path's delay happens to be.
|
||||
Thread.sleep(500)
|
||||
val seen = upstreamCount(control, credential, sessionId)
|
||||
|
||||
val lossPct = if (sent.packets > 0 && seen != null) {
|
||||
round2((sent.packets - seen.packets).coerceAtLeast(0) * 100.0 / sent.packets)
|
||||
} else {
|
||||
null
|
||||
}
|
||||
// The receiver's rate is the measurement. The sender's is what we managed to emit, which
|
||||
// is a property of this phone and its radio, not of the network.
|
||||
val achievedKbps = seen?.kbps ?: 0
|
||||
|
||||
val metrics = json.encodeToJsonElement(
|
||||
UpstreamThroughputMetrics(
|
||||
requestedKbps = kbps,
|
||||
sentPackets = sent.packets,
|
||||
sentBytes = sent.bytes,
|
||||
sentKbps = sent.kbps,
|
||||
receivedPackets = seen?.packets,
|
||||
receivedBytes = seen?.bytes,
|
||||
receivedKbps = achievedKbps,
|
||||
lossPct = lossPct,
|
||||
// Same honesty rule as downstream: if what arrived matches what we offered, the
|
||||
// path was never the constraint and this number says nothing about it.
|
||||
measuresNetwork = seen != null && achievedKbps > 0 && achievedKbps < sent.kbps * 9 / 10,
|
||||
),
|
||||
) as JsonObject
|
||||
|
||||
val findings = ArrayList<Finding>()
|
||||
if (seen != null && seen.packets == 0 && sent.packets > 0) {
|
||||
findings.add(
|
||||
finding(
|
||||
FindingRegistry.THROUGHPUT_NO_DELIVERY, testId,
|
||||
"No upstream traffic reached the server",
|
||||
"This device sent ${sent.packets} packets and the server received none. " +
|
||||
"That is a connectivity fault on the outbound path rather than a slow link.",
|
||||
),
|
||||
)
|
||||
} else if (lossPct != null && lossPct >= 2.0) {
|
||||
findings.add(
|
||||
finding(
|
||||
FindingRegistry.THROUGHPUT_BELOW_OFFERED, testId,
|
||||
"Upstream loss of $lossPct % at ${sent.kbps / 1000} Mbit/s",
|
||||
"The server received ${seen?.packets} of the ${sent.packets} packets this " +
|
||||
"device sent. The outbound path could not carry what was offered.",
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
return Test(
|
||||
id = testId, type = TestType.PERF_THROUGHPUT_UDP, sessionRef = sessionRef, tier = Tier.APP,
|
||||
startedMonoNs = started, endedMonoNs = ids.monoNs(),
|
||||
status = if (seen == null || seen.packets == 0) TestStatus.FAILED else TestStatus.OK,
|
||||
metrics = metrics,
|
||||
) to findings
|
||||
}
|
||||
|
||||
private data class UpstreamCount(val packets: Int, val bytes: Long, val kbps: Int)
|
||||
|
||||
/** The server's tally for this session's upstream run. */
|
||||
private fun upstreamCount(
|
||||
control: ControlClient, credential: String, sessionId: String,
|
||||
): UpstreamCount? = runCatching {
|
||||
val o = Json.parseToJsonElement(control.observations(credential, sessionId))
|
||||
.jsonObject["throughput_up"]?.jsonObject ?: return null
|
||||
UpstreamCount(
|
||||
packets = o["packets"]?.jsonPrimitive?.content?.toIntOrNull() ?: 0,
|
||||
bytes = o["bytes"]?.jsonPrimitive?.content?.toLongOrNull() ?: 0,
|
||||
kbps = o["kbps"]?.jsonPrimitive?.content?.toIntOrNull() ?: 0,
|
||||
)
|
||||
}.getOrNull()
|
||||
|
||||
private data class SenderReport(
|
||||
val packets: Int, val bytes: Long, val kbps: Int, val limitedBy: String,
|
||||
)
|
||||
@@ -169,18 +280,44 @@ class ThroughputMeasurement(private val ids: IdSource) {
|
||||
private fun parseInt(body: String?, key: String): Int? =
|
||||
body?.let { Regex("\"$key\"\\s*:\\s*(-?\\d+)").find(it)?.groupValues?.get(1)?.toIntOrNull() }
|
||||
|
||||
private fun finding(code: String, cat: Category, sev: Severity, testId: String, title: String, desc: String) =
|
||||
/**
|
||||
* Builds a finding from a registry entry, which supplies the code, category and severity.
|
||||
*
|
||||
* Taking a [FindingSpec] rather than three loose values is the point: a typo becomes a
|
||||
* compile error, and two call sites cannot disagree about which category a finding belongs
|
||||
* to - a disagreement that would split one fault across two verdict lights.
|
||||
*/
|
||||
private fun finding(spec: FindingSpec, testId: String, title: String, desc: String) =
|
||||
Finding(
|
||||
id = ids.uuid(), code = code, category = cat, severity = sev, confidence = Confidence.HIGH,
|
||||
id = ids.uuid(), code = spec.code, category = spec.category, severity = spec.severity,
|
||||
confidence = Confidence.HIGH,
|
||||
title = title, description = desc, evidenceRefs = listOf(EvidenceRef(testId)),
|
||||
)
|
||||
|
||||
private fun round2(v: Double) = Math.round(v * 100.0) / 100.0
|
||||
}
|
||||
|
||||
/** Metrics for perf.throughput_udp in the upstream direction. */
|
||||
@Serializable
|
||||
data class UpstreamThroughputMetrics(
|
||||
val direction: String = "up",
|
||||
@SerialName("requested_kbps") val requestedKbps: Int,
|
||||
@SerialName("sent_packets") val sentPackets: Int,
|
||||
@SerialName("sent_bytes") val sentBytes: Long,
|
||||
/** What this device managed to emit — a property of the phone and its radio, not the path. */
|
||||
@SerialName("sent_kbps") val sentKbps: Int,
|
||||
@SerialName("received_packets") val receivedPackets: Int? = null,
|
||||
@SerialName("received_bytes") val receivedBytes: Long? = null,
|
||||
/** What arrived, measured by the only party that can measure it. This is the result. */
|
||||
@SerialName("received_kbps") val receivedKbps: Int,
|
||||
@SerialName("loss_pct") val lossPct: Double? = null,
|
||||
@SerialName("measures_network") val measuresNetwork: Boolean,
|
||||
)
|
||||
|
||||
/** Metrics for perf.throughput_udp. */
|
||||
@Serializable
|
||||
data class ThroughputMetrics(
|
||||
val direction: String = "down",
|
||||
@SerialName("requested_kbps") val requestedKbps: Int,
|
||||
@SerialName("planned_duration_ms") val plannedDurationMs: Int,
|
||||
@SerialName("packets_received") val packetsReceived: Int,
|
||||
|
||||
@@ -65,4 +65,41 @@ class LiveThroughputTest {
|
||||
assertTrue(m.contains(""""limited_by":"duration""""),
|
||||
"the run did not end on the clock, so the rate measures the server, not the path: $m")
|
||||
}
|
||||
|
||||
// Upstream is the direction only the far end can measure. The assertion that matters is that
|
||||
// the server's count is present and plausible against what we sent — a test that only checked
|
||||
// "we transmitted some Mbps" would pass against a server that counted nothing at all.
|
||||
@Test
|
||||
fun measuresUpstreamAgainstTheServersCount() {
|
||||
if (url == null || pin == null || cred == null || udp == null) {
|
||||
println("LiveThroughputTest(up) skipped"); return
|
||||
}
|
||||
val control = ControlClient(url, setOf(pin), "0.2.0")
|
||||
val session = control.createSession(cred, target)
|
||||
val (host, port) = udp.split(":").let { it[0] to it[1].toInt() }
|
||||
|
||||
val (test, findings) = ProbeSession(cred, session, host, port).use { ps ->
|
||||
ps.echo()
|
||||
ThroughputMeasurement(SystemIdSource()).runUpstream(
|
||||
cred, session.sessionId, control, ps, sessionRef = "sess-1",
|
||||
durationS = 3, kbps = 10_000,
|
||||
)
|
||||
}
|
||||
control.deleteSession(cred, session.sessionId)
|
||||
|
||||
val m = assertNotNull(test.metrics).toString()
|
||||
println("upstream: ${test.status} $m")
|
||||
for (f in findings) println("finding ${f.code} [${f.severity}] ${f.title}")
|
||||
|
||||
assertEquals(TestStatus.OK, test.status, "the server counted nothing: $m")
|
||||
val recv = Regex(""""received_packets":(\d+)""").find(m)?.groupValues?.get(1)?.toInt()
|
||||
val sent = Regex(""""sent_packets":(\d+)""").find(m)?.groupValues?.get(1)?.toInt()
|
||||
assertNotNull(recv); assertNotNull(sent)
|
||||
assertTrue(sent > 100, "barely anything was sent, so the rate means nothing: $m")
|
||||
assertTrue(recv > 0, "the server received none of $sent packets: $m")
|
||||
// The counts should be close on a healthy path; wildly different means the two sides are
|
||||
// counting different things rather than the network losing packets.
|
||||
assertTrue(recv <= sent, "the server counted MORE than we sent — the counter is not being reset")
|
||||
println("sent $sent, server saw $recv")
|
||||
}
|
||||
}
|
||||
+205
@@ -0,0 +1,205 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package app.echo_lot.measurement
|
||||
|
||||
/**
|
||||
* The registry of finding codes (measurement-schema.md §9, open item 1).
|
||||
*
|
||||
* A finding code is the stable, machine-readable half of a result: the prose changes, the code is
|
||||
* what a dashboard groups by and what someone greps a year of archived runs for. That only holds
|
||||
* if a code means exactly one thing forever — which is not something ad-hoc string literals at
|
||||
* fifteen call sites can promise.
|
||||
*
|
||||
* The failure this exists to prevent had already happened by the time it was written. Two
|
||||
* independently-added emitters produced `connectivity.downstream_loss` and
|
||||
* `connectivity.loss_downstream` for the same concept, and nothing anywhere objected. Anyone
|
||||
* aggregating either one would have silently seen half their data.
|
||||
*
|
||||
* So codes are declared here as typed specs, each carrying its category and default severity, and
|
||||
* emitters reference the spec rather than retyping the string. That makes a typo a compile error,
|
||||
* and makes it impossible for two call sites to disagree about which category a finding belongs
|
||||
* to — a disagreement that would otherwise split one fault across two verdict lights.
|
||||
*/
|
||||
data class FindingSpec(
|
||||
val code: String,
|
||||
val category: Category,
|
||||
/** Severity when nothing about the specific run argues otherwise; emitters may escalate. */
|
||||
val severity: Severity,
|
||||
/** One line: what this finding asserts. Present tense, no hedging. */
|
||||
val meaning: String,
|
||||
/**
|
||||
* What the finding rules *out*, where that is the useful half. "Loss upstream" is worth much
|
||||
* more when it also says the return path is fine, because that halves where to look next.
|
||||
*/
|
||||
val rulesOut: String? = null,
|
||||
)
|
||||
|
||||
object FindingRegistry {
|
||||
|
||||
// ---- connectivity ----------------------------------------------------------------
|
||||
|
||||
// Renamed from nat.* before anything shipped: neither of these is about NAT, and the
|
||||
// prefix is what decides which category - and therefore which verdict light - a finding
|
||||
// rolls up into. A nat.* code landing under connectivity would be a permanent puzzle.
|
||||
val UDP_UNREACHABLE = FindingSpec(
|
||||
"connectivity.udp_unreachable", Category.CONNECTIVITY, Severity.HIGH,
|
||||
"No UDP echo replies came back from the server at all.",
|
||||
)
|
||||
|
||||
val UDP_UNREACHABLE_UPSTREAM = FindingSpec(
|
||||
"connectivity.udp_unreachable_upstream", Category.CONNECTIVITY, Severity.HIGH,
|
||||
"The server received none of the probes, so traffic is dropped on the way out.",
|
||||
rulesOut = "The return path: nothing arrived to be replied to.",
|
||||
)
|
||||
|
||||
val UDP_LOSS = FindingSpec(
|
||||
"connectivity.udp_loss", Category.CONNECTIVITY, Severity.MEDIUM,
|
||||
"A large fraction of round-trip probes were lost, direction unknown.",
|
||||
)
|
||||
|
||||
val LOSS_UPSTREAM = FindingSpec(
|
||||
"connectivity.loss_upstream", Category.CONNECTIVITY, Severity.MEDIUM,
|
||||
"Probes were lost on the way to the server.",
|
||||
rulesOut = "The return path: replies came back for everything that arrived.",
|
||||
)
|
||||
|
||||
/**
|
||||
* The single code for "lost on the return path", whichever measurement found it.
|
||||
*
|
||||
* Two emitters had independently invented `connectivity.downstream_loss` and
|
||||
* `connectivity.loss_downstream` for this, and nothing objected. Anyone aggregating either
|
||||
* one would have silently seen half their data. Paired with [LOSS_UPSTREAM] so the two
|
||||
* directions read as a set.
|
||||
*/
|
||||
val LOSS_DOWNSTREAM = FindingSpec(
|
||||
"connectivity.loss_downstream", Category.CONNECTIVITY, Severity.MEDIUM,
|
||||
"Packets were lost on the way back from the server.",
|
||||
rulesOut = "The outbound path: the server received what it was answering.",
|
||||
)
|
||||
|
||||
val DOWNSTREAM_BLOCKED = FindingSpec(
|
||||
"connectivity.downstream_blocked", Category.CONNECTIVITY, Severity.HIGH,
|
||||
"Server-initiated packets never arrive, although round trips work.",
|
||||
rulesOut = "Basic reachability: the path forwards replies, just not unsolicited traffic.",
|
||||
)
|
||||
|
||||
val DOWNSTREAM_REORDER = FindingSpec(
|
||||
"connectivity.downstream_reorder", Category.CONNECTIVITY, Severity.LOW,
|
||||
"Downstream packets arrive in a different order than they were sent.",
|
||||
)
|
||||
|
||||
// MEDIUM, not HIGH: a captive portal is a condition to report, not necessarily a fault - on
|
||||
// hotel or cafe wifi it is exactly what should be there, and logging in clears it. NO_INTERNET
|
||||
// is the HIGH one, because nothing the user does locally fixes that. The registry first said
|
||||
// HIGH; the probe emitting it had always said MEDIUM, and the probe was the considered value.
|
||||
val CAPTIVE_PORTAL = FindingSpec(
|
||||
"connectivity.captive_portal", Category.CONNECTIVITY, Severity.MEDIUM,
|
||||
"A captive portal is intercepting connectivity checks.",
|
||||
)
|
||||
|
||||
val NO_INTERNET = FindingSpec(
|
||||
"connectivity.no_internet", Category.CONNECTIVITY, Severity.HIGH,
|
||||
"Android's own connectivity checks fail on this network.",
|
||||
)
|
||||
|
||||
// ---- mtu -------------------------------------------------------------------------
|
||||
|
||||
val MTU_REDUCED_DOWNSTREAM = FindingSpec(
|
||||
"mtu.reduced_downstream", Category.MTU, Severity.LOW,
|
||||
"The downstream path MTU is below the usual 1500 bytes.",
|
||||
)
|
||||
|
||||
val MTU_DOWNSTREAM_BLACKHOLE = FindingSpec(
|
||||
"mtu.downstream_blackhole", Category.MTU, Severity.MEDIUM,
|
||||
"Datagrams above the path MTU are dropped downstream, fragmented or not.",
|
||||
)
|
||||
|
||||
val FRAGMENTS_BLOCKED = FindingSpec(
|
||||
"mtu.fragments_blocked", Category.MTU, Severity.MEDIUM,
|
||||
"IP fragments do not reach this device even when sent in order.",
|
||||
)
|
||||
|
||||
val FRAGMENT_REORDER_SENSITIVE = FindingSpec(
|
||||
"mtu.fragment_reorder_sensitive", Category.MTU, Severity.LOW,
|
||||
"Fragments are delivered in order but dropped when reordered or delayed.",
|
||||
rulesOut = "Fragmentation itself: in-order fragments arrive fine.",
|
||||
)
|
||||
|
||||
// ---- nat -------------------------------------------------------------------------
|
||||
|
||||
val NAT_UDP_REBINDING = FindingSpec(
|
||||
"nat.udp_rebinding", Category.NAT, Severity.MEDIUM,
|
||||
"A NAT remapped the UDP source port mid-flow.",
|
||||
)
|
||||
|
||||
val NAT_SYMMETRIC = FindingSpec(
|
||||
"nat.symmetric", Category.NAT, Severity.MEDIUM,
|
||||
"The NAT assigns a different external port per destination.",
|
||||
)
|
||||
|
||||
// ---- perf ------------------------------------------------------------------------
|
||||
|
||||
val THROUGHPUT_NO_DELIVERY = FindingSpec(
|
||||
"perf.throughput_no_delivery", Category.PERFORMANCE, Severity.HIGH,
|
||||
"No throughput traffic arrived, although the server sent it.",
|
||||
)
|
||||
|
||||
val THROUGHPUT_BELOW_OFFERED = FindingSpec(
|
||||
"perf.throughput_below_offered", Category.PERFORMANCE, Severity.LOW,
|
||||
"Less throughput arrived than the server sent for the whole run.",
|
||||
)
|
||||
|
||||
// ---- dns -------------------------------------------------------------------------
|
||||
|
||||
val DNS_ANSWER_REWRITTEN = FindingSpec(
|
||||
"dns.answer_rewritten", Category.DNS, Severity.HIGH,
|
||||
"A resolver returned an answer that differs from the authoritative record.",
|
||||
)
|
||||
|
||||
val DNS_AUTHORITATIVE_UNREACHABLE = FindingSpec(
|
||||
"dns.authoritative_unreachable", Category.DNS, Severity.MEDIUM,
|
||||
"The canary zone's authoritative server could not be reached.",
|
||||
)
|
||||
|
||||
// ---- v6 ----------------------------------------------------------------------------
|
||||
//
|
||||
// Prefix is `v6.`, matching the test-type registry (v6.brokenness, v6.happy_eyeballs, ...).
|
||||
// These were `ipv6.*` while declaring Category.IPV6, but the prefix map only knows "v6", so
|
||||
// they silently rolled up under connectivity: the third instance of a prefix disagreeing with
|
||||
// its category and quietly moving a fault to a different verdict light.
|
||||
|
||||
val V6_BROKEN = FindingSpec(
|
||||
"v6.broken", Category.IPV6, Severity.MEDIUM,
|
||||
"IPv6 is configured on this network but does not work.",
|
||||
rulesOut = "Absence of IPv6: it is provisioned, it simply fails.",
|
||||
)
|
||||
|
||||
/**
|
||||
* INFO deliberately, and it needs to stay that way.
|
||||
*
|
||||
* Most networks still do not offer IPv6, and that is not a fault. Reporting it as a warning
|
||||
* lights a yellow verdict on a perfectly healthy network, which teaches people to ignore the
|
||||
* light — the one thing a diagnostic must never do.
|
||||
*/
|
||||
val V6_NOT_OFFERED = FindingSpec(
|
||||
"v6.not_offered", Category.IPV6, Severity.INFO,
|
||||
"This network does not offer IPv6.",
|
||||
)
|
||||
|
||||
/** Every registered finding, in declaration order. */
|
||||
val all: List<FindingSpec> = listOf(
|
||||
UDP_UNREACHABLE, UDP_UNREACHABLE_UPSTREAM, UDP_LOSS, LOSS_UPSTREAM, LOSS_DOWNSTREAM,
|
||||
DOWNSTREAM_BLOCKED, DOWNSTREAM_REORDER, CAPTIVE_PORTAL, NO_INTERNET,
|
||||
MTU_REDUCED_DOWNSTREAM, MTU_DOWNSTREAM_BLACKHOLE, FRAGMENTS_BLOCKED,
|
||||
FRAGMENT_REORDER_SENSITIVE,
|
||||
NAT_UDP_REBINDING, NAT_SYMMETRIC,
|
||||
THROUGHPUT_NO_DELIVERY, THROUGHPUT_BELOW_OFFERED,
|
||||
DNS_ANSWER_REWRITTEN, DNS_AUTHORITATIVE_UNREACHABLE,
|
||||
V6_BROKEN, V6_NOT_OFFERED,
|
||||
)
|
||||
|
||||
private val byCode: Map<String, FindingSpec> = all.associateBy { it.code }
|
||||
|
||||
fun byCode(code: String): FindingSpec? = byCode[code]
|
||||
}
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package app.echo_lot.measurement
|
||||
|
||||
import java.io.File
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertTrue
|
||||
import kotlin.test.fail
|
||||
|
||||
/**
|
||||
* Keeps the finding registry honest.
|
||||
*
|
||||
* The interesting test is the last one: it reads `docs/findings-registry.md` and fails when the
|
||||
* document and the code disagree. Documentation that drifts from its implementation is worse than
|
||||
* none, because it still looks authoritative — and a finding registry is precisely the artifact
|
||||
* other people build tooling against.
|
||||
*/
|
||||
class FindingRegistryTest {
|
||||
|
||||
@Test
|
||||
fun codesAreUnique() {
|
||||
val dupes = FindingRegistry.all.groupBy { it.code }.filterValues { it.size > 1 }.keys
|
||||
assertTrue(dupes.isEmpty(), "duplicate finding codes: $dupes")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun everyDeclaredSpecIsInTheAllList() {
|
||||
// Reflection over the object's properties: a spec that is declared but left out of `all`
|
||||
// is invisible to the doc check and to any consumer enumerating the registry.
|
||||
val declared = FindingRegistry::class.java.declaredMethods
|
||||
.filter { it.parameterCount == 0 && it.returnType == FindingSpec::class.java }
|
||||
.mapNotNull { runCatching { it.invoke(FindingRegistry) as FindingSpec }.getOrNull() }
|
||||
.map { it.code }
|
||||
.toSet()
|
||||
val listed = FindingRegistry.all.map { it.code }.toSet()
|
||||
assertEquals(declared, listed, "declared specs and the `all` list disagree")
|
||||
}
|
||||
|
||||
// The prefix decides the category, and the category decides which verdict light the finding
|
||||
// rolls up into. A code whose prefix disagrees with its category silently moves a fault to a
|
||||
// different light — the exact bug that got two codes renamed out of nat.*.
|
||||
@Test
|
||||
fun everyPrefixMatchesItsCategory() {
|
||||
for (spec in FindingRegistry.all) {
|
||||
val fromPrefix = TestType.category(spec.code)
|
||||
assertEquals(
|
||||
fromPrefix, spec.category,
|
||||
"${spec.code} is declared as ${spec.category} but its prefix maps to $fromPrefix",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun codesFollowTheNamingConvention() {
|
||||
val shape = Regex("^[a-z0-9]+\\.[a-z0-9_]+$")
|
||||
for (spec in FindingRegistry.all) {
|
||||
assertTrue(shape.matches(spec.code), "malformed code: ${spec.code}")
|
||||
assertTrue(spec.meaning.isNotBlank(), "${spec.code} has no meaning")
|
||||
assertTrue(
|
||||
spec.meaning.trimEnd().endsWith("."),
|
||||
"${spec.code}'s meaning should be a sentence: '${spec.meaning}'",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// Two near-identical codes are how one fault ends up split across two dashboards. This is a
|
||||
// blunt check — it will not catch every synonym — but it catches the shape that already
|
||||
// happened: the same words in a different order.
|
||||
@Test
|
||||
fun noTwoCodesAreAnagramsOfEachOther() {
|
||||
val normalised = FindingRegistry.all.associate { spec ->
|
||||
spec.code to spec.code.substringAfter('.').split('_').sorted().joinToString("_")
|
||||
}
|
||||
val clashes = normalised.entries.groupBy { it.value }.filterValues { it.size > 1 }
|
||||
if (clashes.isNotEmpty()) {
|
||||
fail("codes differing only in word order: ${clashes.values.map { g -> g.map { it.key } }}")
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theDocumentAndTheRegistryAgree() {
|
||||
val doc = findDoc() ?: run {
|
||||
println("findings-registry.md not found from ${File(".").absolutePath} — skipping")
|
||||
return
|
||||
}
|
||||
val text = doc.readText()
|
||||
|
||||
// Only table rows count as "documented". Prose may legitimately mention a code that no
|
||||
// longer exists — the rules section explains why two were merged — and treating that as
|
||||
// a registry entry would force the document to forget its own history.
|
||||
val documented = text.lines()
|
||||
.filter { it.trimStart().startsWith("|") }
|
||||
.flatMap { row -> Regex("`([a-z0-9]+\\.[a-z0-9_]+)`").findAll(row).map { it.groupValues[1] } }
|
||||
.toSet()
|
||||
val registered = FindingRegistry.all.map { it.code }.toSet()
|
||||
|
||||
val missingFromDoc = registered - documented
|
||||
val missingFromCode = documented - registered
|
||||
assertTrue(
|
||||
missingFromDoc.isEmpty(),
|
||||
"these codes exist in FindingRegistry but not in docs/findings-registry.md: $missingFromDoc",
|
||||
)
|
||||
assertTrue(
|
||||
missingFromCode.isEmpty(),
|
||||
"docs/findings-registry.md documents codes that no longer exist: $missingFromCode",
|
||||
)
|
||||
|
||||
// And the severities must match, or the document is describing a different system.
|
||||
for (spec in FindingRegistry.all) {
|
||||
val row = text.lines().firstOrNull {
|
||||
it.trimStart().startsWith("|") && it.contains("`${spec.code}`")
|
||||
} ?: continue
|
||||
val severity = spec.severity.name.lowercase()
|
||||
assertTrue(
|
||||
row.contains("| $severity |"),
|
||||
"${spec.code} is ${severity} in code but the doc row says otherwise: $row",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/** Walks up from the test's working directory to find the repo's docs/ folder. */
|
||||
private fun findDoc(): File? {
|
||||
var dir: File? = File(".").absoluteFile
|
||||
repeat(6) {
|
||||
val candidate = File(dir, "docs/findings-registry.md")
|
||||
if (candidate.isFile) return candidate
|
||||
dir = dir?.parentFile
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
@@ -20,4 +20,8 @@ kotlin {
|
||||
}
|
||||
java { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 }
|
||||
|
||||
tasks.test { useJUnitPlatform() }
|
||||
tasks.test {
|
||||
useJUnitPlatform()
|
||||
// Opt-in: point this at a captured run to check the anonymizer against real data.
|
||||
System.getenv("ECHOLOT_REAL_RUN")?.let { environment("ECHOLOT_REAL_RUN", it) }
|
||||
}
|
||||
|
||||
@@ -108,12 +108,22 @@ class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
is JsonObject -> walkObject(v, path)
|
||||
is JsonArray -> JsonArray(v.map { walk(key, it, path) })
|
||||
is JsonPrimitive ->
|
||||
if (v.isString) transform(Classification.typeOf(key, path), v.content).let(::JsonPrimitive)
|
||||
else v
|
||||
if (v.isString) {
|
||||
// Name first (it is precise), then shape (it is exhaustive). A field nobody
|
||||
// classified must not be a field that leaks.
|
||||
val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content)
|
||||
JsonPrimitive(transform(type, v.content))
|
||||
} else {
|
||||
v
|
||||
}
|
||||
}
|
||||
|
||||
private fun transform(type: LogicalType?, value: String): String = when (type) {
|
||||
null -> value
|
||||
// Unclassified strings still get their *embedded* identifiers scrubbed. A whole-value
|
||||
// check cannot see them: raw shell output is one long string that is neither a MAC nor an
|
||||
// address, so it sailed through both the name table and the shape check carrying every
|
||||
// MAC on the user's LAN.
|
||||
null -> scrubEmbedded(value)
|
||||
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
|
||||
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
|
||||
LogicalType.IP4 -> ip4(value)
|
||||
@@ -123,6 +133,38 @@ class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
LogicalType.FREETEXT -> "[removed: may contain identifying text]"
|
||||
}
|
||||
|
||||
/**
|
||||
* Replaces addresses and MACs found *inside* a longer string.
|
||||
*
|
||||
* Shizuku probes embed raw command output verbatim — `ip neigh`, `ip route`, `dumpsys` — which
|
||||
* is genuinely valuable evidence and also a complete inventory of every device on the user's
|
||||
* network, with hardware addresses. measurement-schema.md §9 flagged these as "hard to
|
||||
* anonymize" and proposed dropping them from exports.
|
||||
*
|
||||
* Scrubbing beats dropping: the output stays readable and auditable — you can still see the
|
||||
* shape of the neighbour table and how many hosts there were — while the identifiers become
|
||||
* the same pseudonyms used everywhere else in the document. So a MAC appearing both in a
|
||||
* parsed field and in a raw dump still reads as one device.
|
||||
*
|
||||
* Only addresses and MACs are touched, for the same reason as [Classification.inferFromValue]:
|
||||
* they are the patterns that cannot be mistaken for something else in free text.
|
||||
*/
|
||||
private fun scrubEmbedded(value: String): String {
|
||||
// Cheap bail-out: the overwhelming majority of strings are short and contain neither.
|
||||
if (value.length < 7 || (!value.contains(':') && !value.contains('.'))) return value
|
||||
// One pass, not three. Sequential passes re-process their own output: after a MAC became
|
||||
// 78:9a:18:xx:yy:zz the IPv6 pattern matched it — six hex groups separated by colons is
|
||||
// exactly an address — and mangled the vendor prefix that the MAC rule had just taken
|
||||
// care to preserve. Ordered alternation resolves each position once, MAC first.
|
||||
return EMBEDDED.replace(value) { m ->
|
||||
when {
|
||||
m.groups[1] != null -> macPreservingOui(m.value)
|
||||
m.groups[2] != null -> ip6(m.value)
|
||||
else -> ip4(m.value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- per-type transforms -------------------------------------------------------------
|
||||
|
||||
/**
|
||||
@@ -147,6 +189,11 @@ class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
* Public addresses keep only their /16 so the network is still locatable at ISP granularity.
|
||||
*/
|
||||
private fun ip4(value: String): String {
|
||||
// A route destination carries a prefix length; pseudonymize the address and put it back,
|
||||
// or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable.
|
||||
value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len ->
|
||||
return ip4(value.substringBefore('/')) + "/" + len
|
||||
}
|
||||
val o = value.split(".")
|
||||
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
|
||||
val n = o.map { it.toInt() }
|
||||
@@ -167,8 +214,36 @@ class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
* is a device fingerprint, especially with EUI-64.
|
||||
*/
|
||||
private fun ip6(value: String): String {
|
||||
// Dotted quads reach here through the family-agnostic field names (addr, gateway, dst);
|
||||
// hand them to the IPv4 path rather than mangling them as if they were v6.
|
||||
if (value.count { it == ':' } < 2) return ip4(value)
|
||||
if (value.contains('/')) {
|
||||
return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/')
|
||||
}
|
||||
val v = value.lowercase(Locale.ROOT)
|
||||
// The unspecified address and the default route are not identities; mangling them would
|
||||
// make a routing table unreadable for no privacy gain.
|
||||
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
|
||||
|
||||
// Unique local addresses (fc00::/7) need the *whole* prefix replaced, not the tail.
|
||||
//
|
||||
// They look like the v6 equivalent of RFC1918, and the first instinct is to keep them for
|
||||
// the same reason: private, topological, says nothing about anyone. That reasoning does
|
||||
// not carry over. An RFC1918 prefix is shared by millions of networks and identifies
|
||||
// none of them; a ULA global ID is 40 *random* bits, unique to one network by
|
||||
// construction (RFC 4193). It is a network fingerprint. Passing the leading groups
|
||||
// through - which is what the general path does - leaked 32 of those 40 bits.
|
||||
//
|
||||
// The prefix is pseudonymized as a unit, so two addresses on the same ULA subnet still
|
||||
// land on the same pseudonymous prefix. "These hosts are on one network" survives;
|
||||
// "this is *that* network" does not.
|
||||
if (v.startsWith("fc") || v.startsWith("fd")) {
|
||||
val groups = v.substringBefore('%').split(":")
|
||||
val prefix = pseudo("ula-prefix", groups.take(3).joinToString(":")) { it }
|
||||
val host = pseudo("ula-host", v) { it }
|
||||
return "fd${prefix.substring(0, 2)}:${prefix.substring(2, 6)}:${prefix.substring(6, 10)}" +
|
||||
"::${host.substring(0, 4)}"
|
||||
}
|
||||
val groups = v.substringBefore('%').split(":")
|
||||
if (groups.size < 3) return v
|
||||
val h = pseudo("ip6", value) { it }
|
||||
@@ -219,7 +294,7 @@ class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
|
||||
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */
|
||||
private fun pseudo(domain: String, value: String, shape: (String) -> String): String =
|
||||
cache.getOrPut("$domain $value") {
|
||||
cache.getOrPut("$domain\u0000$value") {
|
||||
val md = MessageDigest.getInstance("SHA-256")
|
||||
md.update(salt.bytes)
|
||||
md.update(domain.toByteArray())
|
||||
@@ -229,6 +304,21 @@ class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
}
|
||||
|
||||
private companion object {
|
||||
/**
|
||||
* MAC | IPv6 | IPv4, in that order — alternation is ordered, so a MAC-shaped token is
|
||||
* claimed by the MAC rule before the IPv6 rule can see it.
|
||||
*
|
||||
* The patterns are deliberately conservative. A missed address is scrubbed by another
|
||||
* rule or not at all; an over-eager one mangles timestamps, version strings and log
|
||||
* prefixes, corrupting evidence to protect nothing.
|
||||
*/
|
||||
val EMBEDDED = Regex(
|
||||
// Raw strings: a regex written with escaped escapes is a regex nobody can check.
|
||||
"""(\b[0-9a-fA-F]{2}(?:[:-][0-9a-fA-F]{2}){5}\b)""" +
|
||||
"""|(\b(?:[0-9a-fA-F]{1,4}:){2,7}(?::|[0-9a-fA-F]{1,4})(?:[0-9a-fA-F:]*))""" +
|
||||
"""|(\b(?:\d{1,3}\.){3}\d{1,3}\b)"""
|
||||
)
|
||||
|
||||
val publicSuffixes = setOf(
|
||||
"local", "lan", "home", "internal", "arpa",
|
||||
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk",
|
||||
|
||||
@@ -32,6 +32,16 @@ object Classification {
|
||||
"link_local", "ra_source", "prefix",
|
||||
).forEach { put(it, LogicalType.IP6) }
|
||||
|
||||
// Family-agnostic address fields — the names the models actually use (Address.addr,
|
||||
// Route.gateway, Route.dst, DnsConfig.servers). Their absence here was a real leak: the
|
||||
// device's own global IPv6 address went out verbatim at the level whose description
|
||||
// promises addresses are pseudonymized. Typed IP6 because the transform detects the
|
||||
// family from the value, falling through to the IPv4 path for a dotted quad.
|
||||
listOf(
|
||||
"addr", "address", "gateway", "dst", "src", "servers", "server", "resolver",
|
||||
"next_hop", "via", "public_ip", "observed_ip",
|
||||
).forEach { put(it, LogicalType.IP6) }
|
||||
|
||||
listOf("mac", "hw_addr", "gateway_mac", "router_mac", "sender_mac", "peer_mac")
|
||||
.forEach { put(it, LogicalType.MAC) }
|
||||
listOf("bssid", "ap_mac").forEach { put(it, LogicalType.BSSID) }
|
||||
@@ -40,6 +50,8 @@ object Classification {
|
||||
listOf(
|
||||
"fqdn", "hostname", "host", "name", "reverse_dns", "ptr", "domain", "query_name",
|
||||
"friendly_name", "server_name", "sni", "cname", "search_domain", "device_name",
|
||||
// Plural and prefixed variants the models actually use.
|
||||
"search_domains", "private_dns_hostname", "domains", "hostnames",
|
||||
).forEach { put(it, LogicalType.FQDN) }
|
||||
|
||||
listOf("session_id", "credential", "token", "device_id", "android_id", "serial", "imsi", "iccid")
|
||||
@@ -78,6 +90,49 @@ object Classification {
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Last-resort classification from the *value*, when the field name is unrecognised.
|
||||
*
|
||||
* A name table can only protect fields somebody remembered to add, which is the wrong
|
||||
* property for a privacy control: the dangerous field is the one nobody thought of. This
|
||||
* exists because that failed once already — `addresses[].addr` holds the device's own global
|
||||
* IPv6 address, the table had never heard of the name, and it went out verbatim.
|
||||
*
|
||||
* Only addresses and MACs are inferred, because only those have shapes that cannot be
|
||||
* mistaken for something else. Hostnames deliberately are not: `train.udp_updown` is
|
||||
* indistinguishable from a domain by shape, and mangling a test type would corrupt the
|
||||
* document to protect nothing.
|
||||
*/
|
||||
fun inferFromValue(value: String): LogicalType? {
|
||||
val v = value.trim()
|
||||
if (v.isEmpty() || v.length > 64) return null
|
||||
if (looksLikeMac(v)) return LogicalType.MAC
|
||||
if (looksLikeIp6(v)) return LogicalType.IP6
|
||||
if (looksLikeIp4(v)) return LogicalType.IP4
|
||||
return null
|
||||
}
|
||||
|
||||
private fun isHex(c: Char) = c in '0'..'9' || c in 'a'..'f' || c in 'A'..'F'
|
||||
|
||||
private fun looksLikeMac(v: String): Boolean {
|
||||
val parts = v.split(':', '-')
|
||||
return parts.size == 6 && parts.all { p -> p.length == 2 && p.all(::isHex) }
|
||||
}
|
||||
|
||||
private fun looksLikeIp4(v: String): Boolean {
|
||||
val parts = v.substringBefore('/').split('.')
|
||||
return parts.size == 4 && parts.all { p ->
|
||||
p.isNotEmpty() && p.length <= 3 && p.all(Char::isDigit) && p.toInt() <= 255
|
||||
}
|
||||
}
|
||||
|
||||
private fun looksLikeIp6(v: String): Boolean {
|
||||
val core = v.substringBefore('/').substringBefore('%')
|
||||
// Two colons minimum, so a time or a MAC fragment does not qualify, and nothing but the
|
||||
// characters an address may contain.
|
||||
return core.count { it == ':' } >= 2 && core.all { it == ':' || isHex(it) }
|
||||
}
|
||||
|
||||
fun dropAtBalanced(path: List<String>): Boolean {
|
||||
if (path.isNotEmpty() && path.last() in droppedKeys) return true
|
||||
return droppedPaths.any { dropped -> dropped.all { path.contains(it) } }
|
||||
|
||||
@@ -182,4 +182,47 @@ class AnonymizerTest {
|
||||
assertEquals(PrivacyLevel.BALANCED, PrivacyLevel.max(PrivacyLevel.BALANCED, PrivacyLevel.FULL))
|
||||
assertEquals(PrivacyLevel.FULL, PrivacyLevel.fromWire("nonsense"))
|
||||
}
|
||||
|
||||
// A ULA looks like the v6 RFC1918 and is not. Its global ID is 40 random bits, unique to one
|
||||
// network by construction (RFC 4193), so the prefix IS the identifier - unlike 192.168.x,
|
||||
// which millions of networks share. Passing the leading groups through leaked most of it.
|
||||
@Test
|
||||
fun ulaPrefixesArePseudonymizedWhole() {
|
||||
val doc = json.parseToJsonElement(
|
||||
"""{"run":{"id":"r"},"networks":[{"link":{"dns":{"servers":["fda1:3fb1:ff92:6696::2662"]}}}]}"""
|
||||
).jsonObject
|
||||
val out = flat(anon(PrivacyLevel.BALANCED, doc))
|
||||
assertFalse(out.contains("fda1"), "the ULA global ID survived: $out")
|
||||
assertFalse(out.contains("3fb1"), "part of the ULA global ID survived: $out")
|
||||
assertTrue(out.contains("fd"), "the result should still read as a ULA: $out")
|
||||
}
|
||||
|
||||
// Pseudonymizing the prefix as a unit keeps the one fact that is diagnostically useful:
|
||||
// whether two addresses sit on the same network.
|
||||
@Test
|
||||
fun addressesOnOneUlaSubnetStayRelated() {
|
||||
val doc = json.parseToJsonElement(
|
||||
"""{"run":{"id":"r"},"networks":[{"link":{"dns":{"servers":[
|
||||
"fda1:3fb1:ff92:6696::1","fda1:3fb1:ff92:6696::2","fdff:9999:8888:7777::1"]}}}]}"""
|
||||
).jsonObject
|
||||
val servers = anon(PrivacyLevel.BALANCED, doc)["networks"]!!.jsonArray[0].jsonObject["link"]!!
|
||||
.jsonObject["dns"]!!.jsonObject["servers"]!!.jsonArray.map { it.jsonPrimitive.content }
|
||||
val prefixOf = { s: String -> s.substringBeforeLast("::") }
|
||||
assertEquals(prefixOf(servers[0]), prefixOf(servers[1]),
|
||||
"two addresses on one ULA subnet should share a pseudonymous prefix")
|
||||
assertNotEquals(prefixOf(servers[0]), prefixOf(servers[2]),
|
||||
"a different ULA network must not collide with the first")
|
||||
}
|
||||
|
||||
// RFC1918 stays readable, and this is the contrast that justifies it: a shared, meaningless
|
||||
// prefix is topology; a unique random one is identity.
|
||||
@Test
|
||||
fun rfc1918StaysReadableUnlikeUla() {
|
||||
val doc = json.parseToJsonElement(
|
||||
"""{"run":{"id":"r"},"networks":[{"link":{"dns":{"servers":["192.168.1.1","10.13.102.1"]}}}]}"""
|
||||
).jsonObject
|
||||
val out = flat(anon(PrivacyLevel.BALANCED, doc))
|
||||
assertTrue(out.contains("192.168.1.1"), "RFC1918 should survive: $out")
|
||||
assertTrue(out.contains("10.13.102.1"), "RFC1918 should survive: $out")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package app.echo_lot.privacy
|
||||
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.jsonArray
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import kotlinx.serialization.json.jsonPrimitive
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* The blunt instrument: build a document with identifying values in every place one can actually
|
||||
* occur, anonymize it, and assert none of them survive.
|
||||
*
|
||||
* [AnonymizerTest] checks that the fields the classification table knows about are handled
|
||||
* correctly. This checks the other half — the fields it does *not* know about. A per-field test
|
||||
* can only fail for a field someone remembered to write a case for, which is exactly the wrong
|
||||
* property for a privacy check: the dangerous field is the one nobody thought of.
|
||||
*
|
||||
* Concretely, this is written the way it is because the schema's own field names disagree with
|
||||
* the classifier's. `Address.addr` carries an IP and is documented as such in
|
||||
* measurement-schema.md §8, but the classifier keys on names like `ip4` and `gateway_ip4` and had
|
||||
* never heard of `addr`.
|
||||
*/
|
||||
class LeakTest {
|
||||
|
||||
private val json = Json { prettyPrint = false }
|
||||
private val salt = Salt.perRun(ByteArray(32) { 3 })
|
||||
|
||||
/**
|
||||
* Every string here is something that identifies a person, a household or a device, placed
|
||||
* where the real models actually put it (`core-measurement`'s Network/Link/Address/DnsConfig).
|
||||
*/
|
||||
private val secrets = listOf(
|
||||
"Rambossek WLAN", // ssid
|
||||
"78:9a:18:aa:bb:cc", // bssid
|
||||
"aa:bb:cc:dd:ee:11", // gateway mac
|
||||
"2001:1ad0:c4fe:6767::150", // global v6 address on the interface
|
||||
"2a02:1748:dead:beef::1", // v6 default gateway
|
||||
"203.0.113.77", // public v4
|
||||
"nas.rambossek.lan", // private-dns hostname
|
||||
"rambossek.lan", // search domain
|
||||
"Anna's Chromecast", // neighbour name
|
||||
"kitchen table", // free-text note
|
||||
)
|
||||
|
||||
private fun document(): String = """
|
||||
{
|
||||
"schema": "echolot/measurement",
|
||||
"run": {
|
||||
"id": "run-1", "trigger": "manual", "notes": "${secrets[9]}",
|
||||
"device": {"manufacturer": "OnePlus", "model": "CPH2747"}
|
||||
},
|
||||
"networks": [{
|
||||
"id": "net-1", "transport": "wifi",
|
||||
"link": {
|
||||
"mtu": 1500,
|
||||
"addresses": [
|
||||
{"addr": "${secrets[3]}", "prefix_len": 64, "scope": "global"},
|
||||
{"addr": "192.168.1.44", "prefix_len": 24, "scope": "global"}
|
||||
],
|
||||
"routes": [
|
||||
{"dst": "::/0", "gateway": "${secrets[4]}", "iface": "wlan0"},
|
||||
{"dst": "0.0.0.0/0", "gateway": "192.168.1.1", "iface": "wlan0"}
|
||||
],
|
||||
"dns": {
|
||||
"servers": ["${secrets[5]}", "192.168.1.1"],
|
||||
"private_dns_hostname": "${secrets[6]}",
|
||||
"search_domains": ["${secrets[7]}"]
|
||||
}
|
||||
},
|
||||
"wifi": {"ssid": "${secrets[0]}", "bssid": "${secrets[1]}"},
|
||||
"neighbors": [{"name": "${secrets[8]}", "mac": "${secrets[2]}"}]
|
||||
}],
|
||||
"tests": [{"id": "t1", "type": "train.udp_updown", "status": "ok",
|
||||
"metrics": {"rtt_ms_avg": 12.4}}],
|
||||
"findings": [],
|
||||
"summary": {"verdict": "ok"}
|
||||
}
|
||||
""".trimIndent()
|
||||
|
||||
private fun anonymized(level: PrivacyLevel): String =
|
||||
json.encodeToString(
|
||||
kotlinx.serialization.json.JsonObject.serializer(),
|
||||
Anonymizer(level, salt).anonymize(json.parseToJsonElement(document()).jsonObject),
|
||||
)
|
||||
|
||||
@Test
|
||||
fun nothingIdentifyingSurvivesBalanced() {
|
||||
val out = anonymized(PrivacyLevel.BALANCED)
|
||||
val leaked = secrets.filter { out.contains(it) }
|
||||
assertTrue(
|
||||
leaked.isEmpty(),
|
||||
"these identifying values were uploaded verbatim at BALANCED: $leaked\n\n$out",
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun nothingIdentifyingSurvivesStrict() {
|
||||
val out = anonymized(PrivacyLevel.STRICT)
|
||||
val leaked = secrets.filter { out.contains(it) }
|
||||
assertTrue(leaked.isEmpty(), "leaked at STRICT: $leaked\n\n$out")
|
||||
}
|
||||
|
||||
// Private addresses are kept on purpose — they describe the topology and not the person — so
|
||||
// this pins that the leak test above is not passing by accident of over-redaction.
|
||||
@Test
|
||||
fun privateAddressesAreStillReadable() {
|
||||
val out = anonymized(PrivacyLevel.BALANCED)
|
||||
assertTrue(out.contains("192.168.1.1"), "RFC1918 gateway should survive: $out")
|
||||
assertTrue(out.contains("192.168.1.44"), "RFC1918 interface address should survive: $out")
|
||||
}
|
||||
|
||||
/**
|
||||
* Raw shell output embeds a complete inventory of the local network, and neither the field-name
|
||||
* table nor the whole-value shape check can see it: `ip_neigh` is one long string that is
|
||||
* itself neither a MAC nor an address.
|
||||
*
|
||||
* This is not hypothetical. The blob below is (abridged) real output that reached the server
|
||||
* at the `balanced` level from a test device, carrying the hardware address of every host on
|
||||
* the network. measurement-schema.md §9 had flagged raw dumps as "hard to anonymize"; nothing
|
||||
* enforced it.
|
||||
*/
|
||||
@Test
|
||||
fun identifiersInsideRawShellOutputAreScrubbed() {
|
||||
// Joined rather than written with escapes, so the fixture stays readable and there is no
|
||||
// chance of an escape being mangled on its way into the JSON below.
|
||||
val dump = listOf(
|
||||
"uid=2000",
|
||||
"10.13.102.5 dev wlan0 lladdr 90:09:d0:1a:83:e4 REACHABLE",
|
||||
"10.13.102.1 dev wlan0 lladdr 78:9a:18:54:b8:f9 REACHABLE",
|
||||
"10.13.102.111 dev wlan0 lladdr dc:a2:66:08:69:95 STALE",
|
||||
"2001:4bb8:46a:e724:289d:87ff:feb6:ebd3 dev wlan0 lladdr b8:be:f4:bc:ca:cf STALE",
|
||||
).joinToString(" | ")
|
||||
val doc = json.parseToJsonElement(
|
||||
"""{"run":{"id":"r"},"tests":[{"id":"t","type":"link.ip_monitor",
|
||||
"evidence":{"ip_neigh":"$dump"}}]}"""
|
||||
).jsonObject
|
||||
val out = json.encodeToString(
|
||||
kotlinx.serialization.json.JsonObject.serializer(),
|
||||
Anonymizer(PrivacyLevel.BALANCED, salt).anonymize(doc),
|
||||
)
|
||||
|
||||
for (mac in listOf("90:09:d0:1a:83:e4", "78:9a:18:54:b8:f9", "dc:a2:66:08:69:95", "b8:be:f4:bc:ca:cf")) {
|
||||
assertFalse(out.contains(mac), "a neighbour's MAC survived inside the raw dump: $mac")
|
||||
}
|
||||
assertFalse(out.contains("2001:4bb8:46a:e724:289d:87ff:feb6:ebd3"),
|
||||
"a global IPv6 survived inside the raw dump")
|
||||
|
||||
// Scrubbed, not dropped: the evidence must still be readable, or the raw dump stops being
|
||||
// evidence at all. Structure, hostnames of the fields, and RFC1918 addresses stay.
|
||||
assertTrue(out.contains("REACHABLE") && out.contains("STALE"), "the dump lost its structure")
|
||||
assertTrue(out.contains("10.13.102.1"), "RFC1918 addresses should stay readable: $out")
|
||||
assertTrue(out.contains("78:9a:18"), "the vendor prefix should survive for identification")
|
||||
}
|
||||
|
||||
// A MAC in a raw dump and the same MAC in a parsed field must land on the same pseudonym, or
|
||||
// the document stops being internally consistent and one device reads as two.
|
||||
@Test
|
||||
fun theSameIdentifierMatchesAcrossParsedAndRawFields() {
|
||||
val doc = json.parseToJsonElement(
|
||||
"""{"run":{"id":"r"},
|
||||
"networks":[{"wifi":{"bssid":"78:9a:18:54:b8:f9"}}],
|
||||
"tests":[{"id":"t","evidence":{"ip_neigh":"gw dev wlan0 lladdr 78:9a:18:54:b8:f9 REACHABLE"}}]}"""
|
||||
).jsonObject
|
||||
val out = Anonymizer(PrivacyLevel.BALANCED, salt).anonymize(doc)
|
||||
val parsed = out["networks"]!!.jsonArray[0].jsonObject["wifi"]!!.jsonObject["bssid"]!!
|
||||
.jsonPrimitive.content
|
||||
val raw = json.encodeToString(kotlinx.serialization.json.JsonObject.serializer(), out)
|
||||
assertTrue(raw.contains(parsed),
|
||||
"the parsed BSSID pseudonym ($parsed) does not appear in the scrubbed dump")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package app.echo_lot.privacy
|
||||
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import java.io.File
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Runs the anonymizer over a real captured document when one is supplied via ECHOLOT_REAL_RUN,
|
||||
* and reports every MAC and public address that survives.
|
||||
*
|
||||
* Fixtures only contain the identifiers somebody thought to put in them. A real run off a real
|
||||
* phone contains whatever the probes actually produce — which is how the raw-shell-output leak was
|
||||
* found in the first place. Self-skips when no document is supplied, so nobody's network ends up
|
||||
* committed to the repository.
|
||||
*/
|
||||
class RealDocumentTest {
|
||||
|
||||
@Test
|
||||
fun noIdentifiersSurviveInARealDocument() {
|
||||
val path = System.getenv("ECHOLOT_REAL_RUN")
|
||||
if (path.isNullOrBlank() || !File(path).isFile) {
|
||||
println("RealDocumentTest skipped (set ECHOLOT_REAL_RUN to a captured run)"); return
|
||||
}
|
||||
val json = Json { prettyPrint = false }
|
||||
val doc = json.parseToJsonElement(File(path).readText()).jsonObject
|
||||
val out = json.encodeToString(
|
||||
JsonObject.serializer(),
|
||||
Anonymizer(PrivacyLevel.BALANCED, Salt.perRun(ByteArray(32) { 5 })).anonymize(doc),
|
||||
)
|
||||
|
||||
val macs = Regex("""\b[0-9a-fA-F]{2}(?::[0-9a-fA-F]{2}){5}\b""").findAll(out)
|
||||
.map { it.value.lowercase() }
|
||||
.filter { it != "00:00:00:00:00:00" }
|
||||
.toSet()
|
||||
val original = Regex("""\b[0-9a-fA-F]{2}(?::[0-9a-fA-F]{2}){5}\b""")
|
||||
.findAll(File(path).readText()).map { it.value.lowercase() }.toSet()
|
||||
|
||||
val survived = macs intersect original
|
||||
println("MACs in the original: ${original.size}; unchanged after anonymizing: ${survived.size}")
|
||||
assertTrue(survived.isEmpty(), "these real MAC addresses survived anonymization: $survived")
|
||||
}
|
||||
}
|
||||
@@ -109,6 +109,49 @@ class ProbeSession(
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* Sends paced upstream traffic for [durationMs] and reports what was put on the wire.
|
||||
*
|
||||
* Paced rather than flat out, for the same reason the server paces: an unpaced burst measures
|
||||
* the local NIC and the first queue it meets, then collapses into loss that reads as a network
|
||||
* fault. The schedule is absolute rather than sleep-per-packet, which accumulates the
|
||||
* scheduler's error and drifts the achieved rate below target over a multi-second run.
|
||||
*
|
||||
* Nothing comes back — the server counts and stays silent — so the result here is only the
|
||||
* send side. The measurement is the gap between this and the server's tally.
|
||||
*/
|
||||
fun sendThroughput(durationMs: Long, kbps: Int, sizeBytes: Int = 1200): Sent {
|
||||
val size = sizeBytes.coerceIn(Wire.HEADER_SIZE + 16, 1472)
|
||||
val payload = ByteArray(size - Wire.HEADER_SIZE)
|
||||
val perPacketNs = (size.toLong() * 8 * 1_000_000 / kbps.coerceAtLeast(1)).coerceAtLeast(1_000)
|
||||
|
||||
val start = System.nanoTime()
|
||||
val deadline = start + durationMs * 1_000_000
|
||||
var next = start
|
||||
var packets = 0
|
||||
var bytes = 0L
|
||||
while (System.nanoTime() < deadline) {
|
||||
val pkt = Wire.build(Wire.TYPE_THROUGHPUT_UP, prefix, ++seq, nowNs(), key, payload)
|
||||
try {
|
||||
socket.send(DatagramPacket(pkt, pkt.size, server))
|
||||
} catch (e: java.io.IOException) {
|
||||
// A local send failure is our condition, not the path's. Stop and report what
|
||||
// actually left, rather than counting the remainder as loss on the network.
|
||||
break
|
||||
}
|
||||
packets++
|
||||
bytes += pkt.size
|
||||
next += perPacketNs
|
||||
val sleepNs = next - System.nanoTime()
|
||||
if (sleepNs > 0) Thread.sleep(sleepNs / 1_000_000, (sleepNs % 1_000_000).toInt())
|
||||
}
|
||||
val elapsedMs = (System.nanoTime() - start) / 1_000_000
|
||||
return Sent(packets, bytes, elapsedMs, if (elapsedMs > 0) (bytes * 8 / elapsedMs).toInt() else 0)
|
||||
}
|
||||
|
||||
/** What one upstream run put on the wire locally. */
|
||||
data class Sent(val packets: Int, val bytes: Long, val durationMs: Long, val kbps: Int)
|
||||
|
||||
/** One packet received from the server, with the wire size actually delivered. */
|
||||
data class Received(val type: Int, val seq: Int, val sizeBytes: Int, val tRxNs: Long)
|
||||
|
||||
|
||||
@@ -41,6 +41,13 @@ object Wire {
|
||||
/** One packet of a sustained-rate downstream run. */
|
||||
const val TYPE_THROUGHPUT_DATA: Int = 0x0E
|
||||
|
||||
/**
|
||||
* One packet of a client-driven upstream run. The server counts it and does not answer:
|
||||
* a reply would double the traffic and drag the return path into a measurement that is
|
||||
* specifically about the outbound one.
|
||||
*/
|
||||
const val TYPE_THROUGHPUT_UP: Int = 0x0F
|
||||
|
||||
/** The 8-byte on-the-wire prefix = first 16 hex chars of the session id, decoded. */
|
||||
fun wirePrefix(sessionId: String): ByteArray {
|
||||
require(sessionId.length >= 16) { "session id too short" }
|
||||
|
||||
@@ -22,4 +22,7 @@ VOLUME ["/state"]
|
||||
# the data plane must see real client source addresses/TTLs, and Docker's
|
||||
# userland NAT would falsify exactly what this server exists to observe.
|
||||
EXPOSE 8441/tcp 8442/udp 8443/tcp
|
||||
# The verb is explicit here too, so `docker run <image>` serves and `docker run <image> --help`
|
||||
# still works by overriding the command.
|
||||
ENTRYPOINT ["/echolot-server"]
|
||||
CMD ["--serve"]
|
||||
|
||||
@@ -146,3 +146,41 @@ CI (`.gitea/workflows/build-server.yml`): tests on every push touching `server/`
|
||||
tagging `server-v1.2.3` builds + pushes the container image to the Gitea registry and
|
||||
attaches static linux amd64/arm64 binaries (+ SHA256SUMS) to a release — the same
|
||||
artifacts `--self-update` consumes.
|
||||
|
||||
## TLS for the admin UI
|
||||
|
||||
The binary terminates TLS itself; there is no reverse proxy in the design. It already serves TLS
|
||||
for the control plane, so this is reuse rather than new machinery, and it keeps the "one process,
|
||||
one config file" property. A proxy would also invite someone to eventually front the control plane
|
||||
too — which would break SPKI pinning, because clients pin *that* certificate's key.
|
||||
|
||||
```
|
||||
ECHOLOT_ADMIN_LISTEN=[2001:db8::2]:443
|
||||
ECHOLOT_ADMIN_TLS_CERT=/etc/echolot/admin.pem
|
||||
ECHOLOT_ADMIN_TLS_KEY=/etc/echolot/admin.key
|
||||
ECHOLOT_ADMIN_BASE_URL=https://admin.example.net
|
||||
```
|
||||
|
||||
Certificates come from any ACME client. **DNS-01 is the one to use here**: it needs no inbound
|
||||
port 80, which matters on a host where 80 is awkward or already spoken for.
|
||||
|
||||
```sh
|
||||
acme.sh --issue --dns dns_cf -d admin.example.net \
|
||||
--key-file /etc/echolot/admin.key \
|
||||
--fullchain-file /etc/echolot/admin.pem
|
||||
```
|
||||
|
||||
**No reload hook is needed.** The certificate is re-read when the files change, so a renewal that
|
||||
drops new files in place is picked up on the next handshake. That is deliberate: a reload hook is
|
||||
the part of a renewal setup that quietly stops working, months later, and is noticed only once the
|
||||
certificate has already expired. A torn write — renewal tools write cert and key separately — keeps
|
||||
the previous certificate rather than failing the listener.
|
||||
|
||||
Serving the admin UI in plaintext on a non-loopback address is refused: the session cookie is a
|
||||
bearer credential for everything the server can do, and the OIDC authorization code arrives in a
|
||||
URL. Bind to loopback and use an SSH tunnel (`ssh -L 8444:localhost:8444 host`), supply a
|
||||
certificate, or set `ECHOLOT_ADMIN_INSECURE=1` if you mean it.
|
||||
|
||||
The control-plane certificate is deliberately *not* hot-reloaded. Clients pin its public key, so
|
||||
replacing it is a rotation an operator should have to think about, not something that happens
|
||||
because a file changed.
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
@@ -36,11 +37,15 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"echo-lot.app/server/internal/acmehttp"
|
||||
"echo-lot.app/server/internal/adminauth"
|
||||
"echo-lot.app/server/internal/canarydns"
|
||||
"echo-lot.app/server/internal/certreload"
|
||||
"echo-lot.app/server/internal/compat"
|
||||
"echo-lot.app/server/internal/config"
|
||||
"echo-lot.app/server/internal/control"
|
||||
"echo-lot.app/server/internal/dataplane"
|
||||
"echo-lot.app/server/internal/oidc"
|
||||
"echo-lot.app/server/internal/runs"
|
||||
"echo-lot.app/server/internal/selftest"
|
||||
"echo-lot.app/server/internal/selfupdate"
|
||||
@@ -69,6 +74,30 @@ func run() error {
|
||||
control.Version = Version
|
||||
|
||||
switch {
|
||||
case actions.Help:
|
||||
// Compatibility shim for one release.
|
||||
//
|
||||
// Serving became an explicit verb, but self-update is run by the *old* binary — so the
|
||||
// repair added to the updater cannot fix the very update that installs the new one. A
|
||||
// unit written before this change starts us with no arguments, and without this branch
|
||||
// the service would simply stop working, unattended, on a host nobody is watching.
|
||||
//
|
||||
// Only when systemd started us: INVOCATION_ID is set by systemd for every service
|
||||
// invocation and by nothing else, so a person at a terminal still gets usage. Remove
|
||||
// this once no deployment predates --serve.
|
||||
if os.Getenv("INVOCATION_ID") != "" {
|
||||
slog.Warn("started by systemd with no verb — this unit predates --serve; " +
|
||||
"repairing it and serving anyway")
|
||||
if repaired, err := system.RepairExecStart(); err != nil {
|
||||
slog.Error("could not repair the unit; fix ExecStart by hand", "err", err)
|
||||
} else if repaired {
|
||||
slog.Info("systemd unit updated to pass --serve")
|
||||
}
|
||||
return serve(cfg)
|
||||
}
|
||||
config.Usage(os.Stderr)
|
||||
os.Exit(2)
|
||||
return nil
|
||||
case actions.Version:
|
||||
fmt.Println(Version)
|
||||
return nil
|
||||
@@ -79,6 +108,8 @@ func run() error {
|
||||
return system.InstallSystemd(cfg.SelfUpdateAPI)
|
||||
case actions.UninstallSystemd:
|
||||
return system.UninstallSystemd()
|
||||
case actions.SetAdminPassword:
|
||||
return setAdminPassword(cfg)
|
||||
case actions.SelfUpdate:
|
||||
return selfupdate.Run(cfg.SelfUpdateAPI, Version)
|
||||
}
|
||||
@@ -150,6 +181,29 @@ func serve(cfg *config.Config) error {
|
||||
slog.Info("client compatibility", "accepts_app", appRange.String(),
|
||||
"protocol", control.ProtocolVersion, "schema", control.SchemaVersion)
|
||||
|
||||
// Identity is optional. Without an issuer the server simply has no sign-in, and
|
||||
// uploads=account can never be satisfied — which is the honest outcome, not a silent
|
||||
// downgrade to anonymous.
|
||||
var idp *oidc.Verifier
|
||||
if cfg.OIDCIssuer != "" && (cfg.OIDCClientID != "" || cfg.OIDCAppClientID != "") {
|
||||
idp = oidc.New(oidc.Config{
|
||||
Issuer: cfg.OIDCIssuer,
|
||||
ClientID: cfg.OIDCClientID,
|
||||
AppClientID: cfg.OIDCAppClientID,
|
||||
AdminGroup: cfg.OIDCAdminGroup,
|
||||
}, nil)
|
||||
slog.Info("identity provider configured", "issuer", cfg.OIDCIssuer,
|
||||
"admin_client_id", cfg.OIDCClientID, "app_client_id", cfg.OIDCAppClientID,
|
||||
"admin_group", cfg.OIDCAdminGroup)
|
||||
if cfg.OIDCAdminGroup == "" {
|
||||
slog.Warn("no admin group set: nobody will be an admin via OIDC " +
|
||||
"(set ECHOLOT_OIDC_ADMIN_GROUP)")
|
||||
}
|
||||
} else if cfg.UploadsMode == string(runs.ModeAccount) {
|
||||
slog.Warn("uploads=account but no identity provider is configured — " +
|
||||
"every upload will be refused")
|
||||
}
|
||||
|
||||
ctl := &control.Server{
|
||||
Store: st, Sessions: sessions, Name: cfg.Name,
|
||||
UDPPort: mustPort(firstAddr(cfg.UDPListen)), TCPPort: mustPort(firstAddr(cfg.TCPListen)),
|
||||
@@ -161,6 +215,7 @@ func serve(cfg *config.Config) error {
|
||||
Runs: runStore,
|
||||
AppRange: appRange,
|
||||
PublicControlURL: publicControlURL(cfg),
|
||||
OIDC: idp,
|
||||
}
|
||||
// Left nil when there is no raw socket, so the handler answers "not implemented" with a
|
||||
// reason rather than failing somewhere deeper.
|
||||
@@ -261,7 +316,47 @@ func serve(cfg *config.Config) error {
|
||||
})
|
||||
})
|
||||
adminSrv := &http.Server{Addr: cfg.AdminListen, Handler: admin, ReadHeaderTimeout: 10 * time.Second}
|
||||
if cfg.AdminTLSCert != "" {
|
||||
// Terminated here rather than behind a reverse proxy: this binary already serves TLS for
|
||||
// the control plane, so it is reuse rather than new machinery, and one process with one
|
||||
// config file is the property that makes this pleasant to run. A proxy would also invite
|
||||
// someone to later front the control plane too, which would break SPKI pinning.
|
||||
reloader, err := certreload.New(cfg.AdminTLSCert, cfg.AdminTLSKey)
|
||||
if err != nil {
|
||||
return fmt.Errorf("admin TLS: %w", err)
|
||||
}
|
||||
adminSrv.TLSConfig = reloader.TLSConfig()
|
||||
if exp := reloader.NotAfter(); !exp.IsZero() {
|
||||
slog.Info("admin UI TLS", "listen", cfg.AdminListen, "cert_expires", exp.Format(time.RFC3339))
|
||||
if time.Until(exp) < 14*24*time.Hour {
|
||||
slog.Warn("admin certificate expires soon", "expires", exp.Format(time.RFC3339))
|
||||
}
|
||||
}
|
||||
go func() { errCh <- fmt.Errorf("admin: %w", adminSrv.ListenAndServeTLS("", "")) }()
|
||||
} else {
|
||||
go func() { errCh <- fmt.Errorf("admin: %w", adminSrv.ListenAndServe()) }()
|
||||
}
|
||||
|
||||
// ACME HTTP-01 responder. Permanent rather than started per renewal: nothing binds and
|
||||
// unbinds, so a renewal cannot fail because the port was briefly busy, and the ACME client
|
||||
// needs only write access to a directory instead of the privilege to bind a low port.
|
||||
if cfg.ACMEHTTPListen != "" {
|
||||
webroot := cfg.ACMEWebroot
|
||||
if webroot == "" {
|
||||
webroot = filepath.Join(cfg.StateDir, "acme")
|
||||
}
|
||||
if err := acmehttp.EnsureWebroot(webroot); err != nil {
|
||||
return fmt.Errorf("acme webroot: %w", err)
|
||||
}
|
||||
acmeSrv := &http.Server{
|
||||
Addr: cfg.ACMEHTTPListen,
|
||||
Handler: acmehttp.Handler(webroot, cfg.AdminBaseURL),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
slog.Info("acme http-01 responder", "listen", cfg.ACMEHTTPListen, "webroot", webroot,
|
||||
"redirects_to", cfg.AdminBaseURL)
|
||||
go func() { errCh <- fmt.Errorf("acme-http: %w", acmeSrv.ListenAndServe()) }()
|
||||
}
|
||||
|
||||
// UDP data plane — one socket per configured address. Distinct sockets
|
||||
// (not wildcard) also guarantee responses leave from the address the
|
||||
@@ -493,3 +588,48 @@ func publicControlURL(cfg *config.Config) string {
|
||||
}
|
||||
return "https://" + addr
|
||||
}
|
||||
|
||||
// setAdminPassword stores the break-glass admin credential.
|
||||
//
|
||||
// The password is read from stdin rather than taken as a flag, so it never lands in shell
|
||||
// history, in the process list where any local user can see it, or in a systemd unit. Piping is
|
||||
// still possible for automation:
|
||||
//
|
||||
// printf '%s' "$PW" | echolot-server --set-admin-password --admin-user ops
|
||||
func setAdminPassword(cfg *config.Config) error {
|
||||
st, err := store.Open(cfg.StateDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("state store: %w", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "New password for %q (input is not echoed if this is a terminal): ", cfg.AdminUser)
|
||||
pw, err := readSecret()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(os.Stderr)
|
||||
|
||||
cred, err := adminauth.NewCredential(cfg.AdminUser, pw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := st.SetLocalAdmin(cred); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "Break-glass admin %q set. This account works even when the identity\n"+
|
||||
"provider does not, which is the point of it — treat the password accordingly.\n", cfg.AdminUser)
|
||||
return nil
|
||||
}
|
||||
|
||||
// readSecret reads one line from stdin, without echo where the terminal allows it.
|
||||
func readSecret() (string, error) {
|
||||
restore, _ := system.DisableEcho(os.Stdin)
|
||||
if restore != nil {
|
||||
defer restore()
|
||||
}
|
||||
r := bufio.NewReader(os.Stdin)
|
||||
line, err := r.ReadString('\n')
|
||||
if err != nil && line == "" {
|
||||
return "", err
|
||||
}
|
||||
return strings.TrimSpace(line), nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
// Package acmehttp answers ACME HTTP-01 challenges and sends everything else to HTTPS.
|
||||
//
|
||||
// HTTP-01 validation always arrives on port 80 — the CA chooses the port, not the operator — so
|
||||
// it never collides with an admin UI on 443. That leaves two ways to answer it: let the ACME
|
||||
// client bind port 80 for a few seconds during each renewal, or keep something there permanently
|
||||
// that serves the challenge directory. This is the second, and it is the better trade:
|
||||
//
|
||||
// - nothing binds and unbinds, so renewal cannot fail because the port was briefly busy;
|
||||
// - the ACME client needs no privileges to bind a low port, only write access to a directory;
|
||||
// - port 80 gets a use it would want anyway, redirecting people who typed http:// to the real
|
||||
// thing instead of hanging.
|
||||
//
|
||||
// It is the same arrangement as the webroot plugins for Apache and nginx, and it works with any
|
||||
// ACME client that can write a file: `lego --http.webroot`, `certbot --webroot`, `acme.sh -w`.
|
||||
//
|
||||
// The ACME client stays an external program on purpose. lego is also a Go library, but importing
|
||||
// it would put a large dependency tree into a server that deliberately has none — and the CLI does
|
||||
// the same job from a timer.
|
||||
package acmehttp
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ChallengePath is the fixed prefix ACME uses. It is not configurable, by the specification.
|
||||
const ChallengePath = "/.well-known/acme-challenge/"
|
||||
|
||||
// Handler serves challenge tokens from webroot and redirects everything else to redirectTo.
|
||||
//
|
||||
// webroot is the directory an ACME client writes into; the tokens themselves land in
|
||||
// <webroot>/.well-known/acme-challenge/<token>, which is exactly what --http.webroot expects.
|
||||
func Handler(webroot, redirectTo string) http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
|
||||
mux.HandleFunc(ChallengePath, func(w http.ResponseWriter, r *http.Request) {
|
||||
token := strings.TrimPrefix(r.URL.Path, ChallengePath)
|
||||
// Tokens are base64url from the CA. Anything else is somebody probing, and refusing by
|
||||
// shape means path traversal never gets as far as touching the filesystem.
|
||||
if token == "" || !validToken(token) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
body, err := os.ReadFile(filepath.Join(webroot, filepath.FromSlash(ChallengePath), token))
|
||||
if err != nil {
|
||||
// Logged at info: a challenge that cannot be answered is why a renewal failed, and
|
||||
// that is worth being able to see afterwards rather than guessing at it.
|
||||
slog.Info("acme challenge not found", "token", token, "webroot", webroot)
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
slog.Info("answered acme challenge", "token", token, "from", r.RemoteAddr)
|
||||
w.Header().Set("Content-Type", "text/plain")
|
||||
_, _ = w.Write(body)
|
||||
})
|
||||
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
if redirectTo == "" {
|
||||
http.Error(w, "this port serves ACME challenges only", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
// 308 rather than 302: the method must not change, and the redirect is permanent in the
|
||||
// sense that matters — this port will never serve the application.
|
||||
http.Redirect(w, r, strings.TrimRight(redirectTo, "/")+r.URL.RequestURI(), http.StatusPermanentRedirect)
|
||||
})
|
||||
|
||||
return mux
|
||||
}
|
||||
|
||||
// validToken accepts only the base64url alphabet the ACME spec uses for tokens.
|
||||
//
|
||||
// A shape check rather than a path check: "../../etc/shadow" fails here before any filesystem
|
||||
// call, which is a stronger guarantee than sanitising a path and hoping the sanitiser is right.
|
||||
func validToken(s string) bool {
|
||||
if len(s) > 128 {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_', r == '.':
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
// A bare "." or ".." never appears in a real token and is the one traversal the alphabet
|
||||
// above would otherwise permit.
|
||||
return s != "." && s != ".."
|
||||
}
|
||||
|
||||
// EnsureWebroot creates the challenge directory, so an ACME client's first run does not fail on a
|
||||
// missing path and an operator does not have to know the layout.
|
||||
func EnsureWebroot(webroot string) error {
|
||||
return os.MkdirAll(filepath.Join(webroot, filepath.FromSlash(ChallengePath)), 0o755)
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package acmehttp
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func serve(t *testing.T, redirectTo string) (http.Handler, string) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
if err := EnsureWebroot(root); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return Handler(root, redirectTo), root
|
||||
}
|
||||
|
||||
func TestServesAChallengeTokenWrittenByAnAcmeClient(t *testing.T) {
|
||||
h, root := serve(t, "https://admin.example.net")
|
||||
// Exactly what `lego --http.webroot` writes.
|
||||
token := "abc-123_XYZ"
|
||||
want := "abc-123_XYZ.keyauthorization-part"
|
||||
if err := os.WriteFile(filepath.Join(root, ".well-known", "acme-challenge", token), []byte(want), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, httptest.NewRequest("GET", ChallengePath+token, nil))
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("challenge not served: %d", rec.Code)
|
||||
}
|
||||
if rec.Body.String() != want {
|
||||
t.Fatalf("body = %q, want %q", rec.Body.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// The token comes from the network and is used to build a path. Rejecting by *shape* means
|
||||
// traversal never reaches the filesystem at all, which is a stronger guarantee than sanitising.
|
||||
func TestTraversalNeverTouchesTheFilesystem(t *testing.T) {
|
||||
h, root := serve(t, "https://admin.example.net")
|
||||
secret := filepath.Join(filepath.Dir(root), "secret.txt")
|
||||
if err := os.WriteFile(secret, []byte("do not serve me"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, bad := range []string{
|
||||
"../secret.txt",
|
||||
"..%2Fsecret.txt",
|
||||
"../../etc/passwd",
|
||||
"..",
|
||||
".",
|
||||
"a/b",
|
||||
"tok%20en", // a space arrives percent-encoded; a literal one is not a valid request line
|
||||
} {
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, httptest.NewRequest("GET", ChallengePath+bad, nil))
|
||||
if rec.Code == 200 && rec.Body.String() == "do not serve me" {
|
||||
t.Fatalf("served a file outside the challenge directory via %q", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEverythingElseRedirectsToHTTPS(t *testing.T) {
|
||||
h, _ := serve(t, "https://admin.example.net")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, httptest.NewRequest("GET", "/devices?page=2", nil))
|
||||
if rec.Code != http.StatusPermanentRedirect {
|
||||
t.Fatalf("code = %d, want 308", rec.Code)
|
||||
}
|
||||
// The path and query must survive, or a bookmarked link lands on the wrong page.
|
||||
if got := rec.Header().Get("Location"); got != "https://admin.example.net/devices?page=2" {
|
||||
t.Fatalf("Location = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// With no admin URL configured there is nowhere to send people, and inventing one would be worse
|
||||
// than saying so.
|
||||
func TestNoRedirectTargetIsHonest(t *testing.T) {
|
||||
h, _ := serve(t, "")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, httptest.NewRequest("GET", "/", nil))
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("code = %d, want 404", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMissingTokenIsNotFound(t *testing.T) {
|
||||
h, _ := serve(t, "https://admin.example.net")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, httptest.NewRequest("GET", ChallengePath+"never-written", nil))
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("code = %d, want 404", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
// Package adminauth handles who may administer the server.
|
||||
//
|
||||
// Two ways in, deliberately:
|
||||
//
|
||||
// - **OIDC**, the normal one. Identity lives in the operator's own IdP.
|
||||
// - **A local admin password**, the break-glass one. If the IdP is misconfigured, unreachable,
|
||||
// or the operator fat-fingered the admin group, they would otherwise be locked out of their
|
||||
// own server with no way back in short of editing JSON on disk. A fallback that only works
|
||||
// when everything else is broken is exactly the thing you cannot add later, because by then
|
||||
// you cannot get in to add it.
|
||||
//
|
||||
// The local password is stored as PBKDF2-HMAC-SHA256, from the standard library (Go 1.24+), with
|
||||
// a per-credential salt. Not because password login is encouraged — it is the fallback — but
|
||||
// because a break-glass credential is precisely the one most likely to end up in a backup or a
|
||||
// config-management repo, and a hash survives that where a bearer token does not.
|
||||
//
|
||||
// There is no email reset flow and there should not be: `--set-admin-password` on the host *is*
|
||||
// the reset, and anyone who can run it already has the machine.
|
||||
package adminauth
|
||||
|
||||
import (
|
||||
"crypto/hmac"
|
||||
"crypto/pbkdf2"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// iterations follows OWASP's guidance for PBKDF2-HMAC-SHA256. Deliberately slow: this credential
|
||||
// is used a handful of times in a server's life, so the cost is invisible to the operator and
|
||||
// meaningful to anyone grinding a stolen hash.
|
||||
const iterations = 600_000
|
||||
|
||||
const (
|
||||
saltLen = 16
|
||||
keyLen = 32
|
||||
)
|
||||
|
||||
// Credential is a stored local admin password.
|
||||
type Credential struct {
|
||||
Username string `json:"username"`
|
||||
Salt string `json:"salt"` // hex
|
||||
Hash string `json:"hash"` // hex
|
||||
Iterations int `json:"iterations"`
|
||||
Updated string `json:"updated,omitempty"`
|
||||
}
|
||||
|
||||
// NewCredential derives a stored credential from a plaintext password.
|
||||
func NewCredential(username, password string) (Credential, error) {
|
||||
if strings.TrimSpace(username) == "" {
|
||||
return Credential{}, errors.New("username must not be empty")
|
||||
}
|
||||
// Twelve is not a policy so much as a floor: this is the one account that can reach
|
||||
// everything, and it is not rate-limited by a human being's patience.
|
||||
if len(password) < 12 {
|
||||
return Credential{}, errors.New("password must be at least 12 characters")
|
||||
}
|
||||
salt := make([]byte, saltLen)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return Credential{}, err
|
||||
}
|
||||
key, err := pbkdf2.Key(sha256.New, password, salt, iterations, keyLen)
|
||||
if err != nil {
|
||||
return Credential{}, err
|
||||
}
|
||||
return Credential{
|
||||
Username: username,
|
||||
Salt: hex.EncodeToString(salt),
|
||||
Hash: hex.EncodeToString(key),
|
||||
Iterations: iterations,
|
||||
Updated: time.Now().UTC().Format(time.RFC3339),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Verify checks a username and password against this credential.
|
||||
//
|
||||
// Both comparisons are constant-time, including the username: a fast rejection on an unknown
|
||||
// username is a timing oracle for which usernames exist. The stored iteration count is used
|
||||
// rather than the current constant, so raising the constant does not lock out existing passwords.
|
||||
func (c Credential) Verify(username, password string) bool {
|
||||
if c.Username == "" || c.Hash == "" {
|
||||
return false
|
||||
}
|
||||
salt, err := hex.DecodeString(c.Salt)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
want, err := hex.DecodeString(c.Hash)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
iter := c.Iterations
|
||||
if iter <= 0 {
|
||||
iter = iterations
|
||||
}
|
||||
got, err := pbkdf2.Key(sha256.New, password, salt, iter, len(want))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
userOK := subtle.ConstantTimeCompare([]byte(c.Username), []byte(username)) == 1
|
||||
passOK := subtle.ConstantTimeCompare(got, want) == 1
|
||||
return userOK && passOK
|
||||
}
|
||||
|
||||
// Throttle slows repeated failures against the local password.
|
||||
//
|
||||
// The local admin is a single well-known account guarding everything, so an unthrottled login
|
||||
// form is an offline-speed guessing oracle that happens to be online. This is deliberately crude
|
||||
// — a delay that grows with consecutive failures and resets on success — because the goal is to
|
||||
// make guessing impractical, not to build a lockout system that an operator can trap themselves
|
||||
// with. It never locks permanently: a break-glass credential that can be locked out by an
|
||||
// attacker is a denial of service against the person who needs it most.
|
||||
type Throttle struct {
|
||||
mu sync.Mutex
|
||||
failures int
|
||||
last time.Time
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
func NewThrottle() *Throttle { return &Throttle{now: time.Now} }
|
||||
|
||||
// Delay is how long the caller should wait before answering, given the failures so far.
|
||||
func (t *Throttle) Delay() time.Duration {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
// A quiet minute forgives everything, so an operator returning later is not punished for
|
||||
// somebody else's earlier attempts.
|
||||
if !t.last.IsZero() && t.now().Sub(t.last) > time.Minute {
|
||||
t.failures = 0
|
||||
}
|
||||
switch {
|
||||
case t.failures == 0:
|
||||
return 0
|
||||
case t.failures < 3:
|
||||
return 250 * time.Millisecond
|
||||
case t.failures < 6:
|
||||
return time.Second
|
||||
default:
|
||||
return 3 * time.Second
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Throttle) Failed() {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.failures++
|
||||
t.last = t.now()
|
||||
}
|
||||
|
||||
func (t *Throttle) Succeeded() {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.failures = 0
|
||||
}
|
||||
|
||||
// ---- sessions ---------------------------------------------------------------------------
|
||||
|
||||
// Session is an authenticated admin, however they proved it.
|
||||
type Session struct {
|
||||
// Subject is the account id: "local:<username>" or "<issuer>#<sub>" from OIDC.
|
||||
Subject string
|
||||
// Display is what the UI shows.
|
||||
Display string
|
||||
Expires time.Time
|
||||
}
|
||||
|
||||
// Sessions mints and checks signed session cookies.
|
||||
//
|
||||
// The cookie carries its own contents and a MAC, so there is no server-side session table to
|
||||
// grow, expire, or lose on restart — and equally no way to revoke one early, which is why they
|
||||
// are short-lived. The secret is persisted, so an operator's session survives a service restart;
|
||||
// regenerating it (deleting it from the state file) invalidates every session at once, which is
|
||||
// the revocation mechanism.
|
||||
type Sessions struct {
|
||||
secret []byte
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
func NewSessions(secret []byte, ttl time.Duration) *Sessions {
|
||||
if ttl <= 0 {
|
||||
ttl = 12 * time.Hour
|
||||
}
|
||||
return &Sessions{secret: append([]byte(nil), secret...), ttl: ttl}
|
||||
}
|
||||
|
||||
// NewSecret makes a fresh signing secret for first start.
|
||||
func NewSecret() ([]byte, error) {
|
||||
b := make([]byte, 32)
|
||||
_, err := rand.Read(b)
|
||||
return b, err
|
||||
}
|
||||
|
||||
var ErrSession = errors.New("session is not valid")
|
||||
|
||||
// Issue returns the cookie value for a newly authenticated admin.
|
||||
func (s *Sessions) Issue(subject, display string) string {
|
||||
exp := time.Now().Add(s.ttl).Unix()
|
||||
payload := base64.RawURLEncoding.EncodeToString([]byte(subject)) + "." +
|
||||
base64.RawURLEncoding.EncodeToString([]byte(display)) + "." +
|
||||
strconv.FormatInt(exp, 10)
|
||||
return payload + "." + s.mac(payload)
|
||||
}
|
||||
|
||||
// Parse checks a cookie value and returns the session it encodes.
|
||||
func (s *Sessions) Parse(value string) (*Session, error) {
|
||||
i := strings.LastIndex(value, ".")
|
||||
if i < 0 {
|
||||
return nil, ErrSession
|
||||
}
|
||||
payload, sig := value[:i], value[i+1:]
|
||||
// MAC first, always. Nothing in the payload is believed — not even its shape — before the
|
||||
// signature has been checked.
|
||||
if !hmac.Equal([]byte(sig), []byte(s.mac(payload))) {
|
||||
return nil, ErrSession
|
||||
}
|
||||
parts := strings.Split(payload, ".")
|
||||
if len(parts) != 3 {
|
||||
return nil, ErrSession
|
||||
}
|
||||
subject, err := base64.RawURLEncoding.DecodeString(parts[0])
|
||||
if err != nil {
|
||||
return nil, ErrSession
|
||||
}
|
||||
display, err := base64.RawURLEncoding.DecodeString(parts[1])
|
||||
if err != nil {
|
||||
return nil, ErrSession
|
||||
}
|
||||
exp, err := strconv.ParseInt(parts[2], 10, 64)
|
||||
if err != nil {
|
||||
return nil, ErrSession
|
||||
}
|
||||
if time.Now().After(time.Unix(exp, 0)) {
|
||||
return nil, fmt.Errorf("%w: expired", ErrSession)
|
||||
}
|
||||
return &Session{Subject: string(subject), Display: string(display), Expires: time.Unix(exp, 0)}, nil
|
||||
}
|
||||
|
||||
func (s *Sessions) mac(payload string) string {
|
||||
m := hmac.New(sha256.New, s.secret)
|
||||
m.Write([]byte(payload))
|
||||
return base64.RawURLEncoding.EncodeToString(m.Sum(nil))
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package adminauth
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PBKDF2 at 600k iterations is slow on purpose, so these use a reduced count where the test is
|
||||
// about logic rather than cost.
|
||||
func fastCredential(t *testing.T, user, pass string) Credential {
|
||||
t.Helper()
|
||||
c, err := NewCredential(user, pass)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func TestVerifyAcceptsOnlyTheRightPair(t *testing.T) {
|
||||
c := fastCredential(t, "admin", "correct-horse-battery")
|
||||
|
||||
if !c.Verify("admin", "correct-horse-battery") {
|
||||
t.Fatal("the correct credentials were rejected")
|
||||
}
|
||||
for _, tc := range []struct{ user, pass string }{
|
||||
{"admin", "wrong-password-here"},
|
||||
{"admin", ""},
|
||||
{"root", "correct-horse-battery"},
|
||||
{"", "correct-horse-battery"},
|
||||
{"ADMIN", "correct-horse-battery"}, // usernames are not case-folded
|
||||
} {
|
||||
if c.Verify(tc.user, tc.pass) {
|
||||
t.Errorf("accepted %q/%q", tc.user, tc.pass)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Two credentials with the same password must not share a hash, or one cracked password reveals
|
||||
// every reuse of it and a precomputed table works against all of them.
|
||||
func TestSaltsDiffer(t *testing.T) {
|
||||
a := fastCredential(t, "admin", "the-same-password-x")
|
||||
b := fastCredential(t, "admin", "the-same-password-x")
|
||||
if a.Salt == b.Salt {
|
||||
t.Fatal("two credentials share a salt")
|
||||
}
|
||||
if a.Hash == b.Hash {
|
||||
t.Fatal("the same password produced the same hash twice")
|
||||
}
|
||||
// Both must still verify — a salt that is not actually used would also produce differing
|
||||
// hashes if it were mixed in wrongly.
|
||||
if !a.Verify("admin", "the-same-password-x") || !b.Verify("admin", "the-same-password-x") {
|
||||
t.Fatal("a salted credential does not verify")
|
||||
}
|
||||
}
|
||||
|
||||
// The stored iteration count is used rather than the current constant, so raising the constant
|
||||
// later does not silently lock out every existing password.
|
||||
func TestOldIterationCountsStillVerify(t *testing.T) {
|
||||
c := fastCredential(t, "admin", "a-perfectly-fine-pw")
|
||||
c.Iterations = iterations // as stored
|
||||
if !c.Verify("admin", "a-perfectly-fine-pw") {
|
||||
t.Fatal("credential does not verify with its stored iteration count")
|
||||
}
|
||||
// A credential written before the field existed must not be treated as zero-iteration.
|
||||
c.Iterations = 0
|
||||
if !c.Verify("admin", "a-perfectly-fine-pw") {
|
||||
t.Fatal("a credential with no recorded iteration count failed to verify")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWeakInputsAreRefusedAtCreation(t *testing.T) {
|
||||
if _, err := NewCredential("", "long-enough-password"); err == nil {
|
||||
t.Error("an empty username was accepted")
|
||||
}
|
||||
if _, err := NewCredential("admin", "short"); err == nil {
|
||||
t.Error("a short password was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEmptyCredentialNeverVerifies(t *testing.T) {
|
||||
var zero Credential
|
||||
if zero.Verify("", "") {
|
||||
t.Fatal("a server with no local admin configured accepted empty credentials")
|
||||
}
|
||||
if zero.Verify("admin", "anything") {
|
||||
t.Fatal("an unset credential verified")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- sessions ----------------------------------------------------------------------------
|
||||
|
||||
func TestSessionRoundTrip(t *testing.T) {
|
||||
secret, _ := NewSecret()
|
||||
s := NewSessions(secret, time.Hour)
|
||||
got, err := s.Parse(s.Issue("local:admin", "Admin"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Subject != "local:admin" || got.Display != "Admin" {
|
||||
t.Fatalf("session did not round-trip: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The cookie carries its own contents, so the MAC is the only thing standing between a user and
|
||||
// promoting themselves. Every tampered form must fail.
|
||||
func TestTamperedSessionsAreRejected(t *testing.T) {
|
||||
secret, _ := NewSecret()
|
||||
s := NewSessions(secret, time.Hour)
|
||||
good := s.Issue("local:admin", "Admin")
|
||||
|
||||
parts := strings.Split(good, ".")
|
||||
tampered := []string{
|
||||
"",
|
||||
"garbage",
|
||||
good + "x", // signature altered
|
||||
strings.Replace(good, parts[0], "Zm9v", 1), // subject swapped
|
||||
strings.Join(parts[:len(parts)-1], "."), // signature removed
|
||||
parts[0] + "." + parts[1] + "." + parts[2], // signature removed, well-formed payload
|
||||
}
|
||||
for _, v := range tampered {
|
||||
if _, err := s.Parse(v); err == nil {
|
||||
t.Errorf("accepted a tampered session: %q", v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionsFromAnotherSecretAreRejected(t *testing.T) {
|
||||
a, _ := NewSecret()
|
||||
b, _ := NewSecret()
|
||||
issued := NewSessions(a, time.Hour).Issue("local:admin", "Admin")
|
||||
if _, err := NewSessions(b, time.Hour).Parse(issued); err == nil {
|
||||
t.Fatal("a session signed with a different secret was accepted — rotating the secret " +
|
||||
"must invalidate every existing session")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExpiredSessionsAreRejected(t *testing.T) {
|
||||
secret, _ := NewSecret()
|
||||
// A negative TTL is not reachable through NewSessions, so issue with a real one and check
|
||||
// the boundary via a session that has already run out.
|
||||
s := NewSessions(secret, time.Millisecond)
|
||||
v := s.Issue("local:admin", "Admin")
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
if _, err := s.Parse(v); err == nil {
|
||||
t.Fatal("an expired session was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- throttle ------------------------------------------------------------------------------
|
||||
|
||||
func TestThrottleGrowsWithFailuresAndResetsOnSuccess(t *testing.T) {
|
||||
tr := NewThrottle()
|
||||
if d := tr.Delay(); d != 0 {
|
||||
t.Fatalf("a first attempt was delayed by %v", d)
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
tr.Failed()
|
||||
}
|
||||
first := tr.Delay()
|
||||
for i := 0; i < 6; i++ {
|
||||
tr.Failed()
|
||||
}
|
||||
later := tr.Delay()
|
||||
if !(later > first && first > 0) {
|
||||
t.Fatalf("delay did not grow with failures: %v then %v", first, later)
|
||||
}
|
||||
tr.Succeeded()
|
||||
if d := tr.Delay(); d != 0 {
|
||||
t.Fatalf("a successful login did not clear the throttle: %v", d)
|
||||
}
|
||||
}
|
||||
|
||||
// A break-glass credential that an attacker can lock out is a denial of service against the one
|
||||
// person who needs it. The delay must stay bounded rather than becoming a lockout.
|
||||
func TestThrottleNeverLocksOutPermanently(t *testing.T) {
|
||||
tr := NewThrottle()
|
||||
for i := 0; i < 1000; i++ {
|
||||
tr.Failed()
|
||||
}
|
||||
if d := tr.Delay(); d > 10*time.Second {
|
||||
t.Fatalf("throttle became a lockout: %v", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestThrottleForgivesAfterAQuietPeriod(t *testing.T) {
|
||||
tr := NewThrottle()
|
||||
now := time.Now()
|
||||
tr.now = func() time.Time { return now }
|
||||
for i := 0; i < 10; i++ {
|
||||
tr.Failed()
|
||||
}
|
||||
if tr.Delay() == 0 {
|
||||
t.Fatal("failures did not register")
|
||||
}
|
||||
now = now.Add(2 * time.Minute)
|
||||
if d := tr.Delay(); d != 0 {
|
||||
t.Fatalf("an operator returning later was still throttled: %v", d)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
// Package certreload serves a TLS certificate that can be replaced on disk without a restart.
|
||||
//
|
||||
// The hard part of TLS is never termination — the stdlib does that — it is renewal. A certificate
|
||||
// obtained from an ACME client expires every sixty days, and the usual arrangement is a renewal
|
||||
// hook that reloads or restarts the service. That hook is the part that quietly fails: it works
|
||||
// when it is written and then, months later, does not, and nobody notices until the certificate
|
||||
// has already expired.
|
||||
//
|
||||
// So the certificate is re-read when the file changes. There is no hook to forget, no reload to
|
||||
// coordinate, and a renewal that drops new files in place is picked up on the next handshake.
|
||||
//
|
||||
// Deliberately not used for the control plane. Clients pin that certificate's public key
|
||||
// (probe-protocol.md §1), so swapping it at runtime would silently break every enrolled device —
|
||||
// there the operator *should* have to think, and a restart is the least of what a key rotation
|
||||
// costs. Two listeners, two different right answers.
|
||||
package certreload
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"fmt"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Reloader holds a certificate and refreshes it when the files on disk change.
|
||||
type Reloader struct {
|
||||
certPath, keyPath string
|
||||
|
||||
mu sync.RWMutex
|
||||
cert *tls.Certificate
|
||||
certMod time.Time
|
||||
keyMod time.Time
|
||||
checked time.Time
|
||||
interval time.Duration
|
||||
}
|
||||
|
||||
// New loads the pair once so a bad path fails at startup rather than at the first handshake,
|
||||
// when the only symptom is a connection error at the far end.
|
||||
func New(certPath, keyPath string) (*Reloader, error) {
|
||||
r := &Reloader{certPath: certPath, keyPath: keyPath, interval: 30 * time.Second}
|
||||
if err := r.load(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// TLSConfig returns a config that asks this reloader for the certificate on every handshake.
|
||||
func (r *Reloader) TLSConfig() *tls.Config {
|
||||
return &tls.Config{
|
||||
MinVersion: tls.VersionTLS12,
|
||||
GetCertificate: r.getCertificate,
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Reloader) getCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) {
|
||||
r.maybeReload()
|
||||
r.mu.RLock()
|
||||
defer r.mu.RUnlock()
|
||||
if r.cert == nil {
|
||||
return nil, fmt.Errorf("no certificate loaded")
|
||||
}
|
||||
return r.cert, nil
|
||||
}
|
||||
|
||||
// maybeReload stats the files at most once per interval.
|
||||
//
|
||||
// Rate-limited because this runs on every handshake: a busy listener would otherwise stat twice
|
||||
// per connection, and a certificate that is thirty seconds stale has never mattered to anyone.
|
||||
func (r *Reloader) maybeReload() {
|
||||
r.mu.RLock()
|
||||
fresh := time.Since(r.checked) < r.interval
|
||||
r.mu.RUnlock()
|
||||
if fresh {
|
||||
return
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
r.checked = time.Now()
|
||||
certMod, keyMod := modTime(r.certPath), modTime(r.keyPath)
|
||||
unchanged := certMod.Equal(r.certMod) && keyMod.Equal(r.keyMod)
|
||||
r.mu.Unlock()
|
||||
if unchanged {
|
||||
return
|
||||
}
|
||||
// A failed reload keeps the certificate already in memory. Renewal tools write the two files
|
||||
// separately, so there is a window where the pair does not match; serving the previous
|
||||
// certificate through that window is strictly better than serving none.
|
||||
_ = r.load()
|
||||
}
|
||||
|
||||
func (r *Reloader) load() error {
|
||||
cert, err := tls.LoadX509KeyPair(r.certPath, r.keyPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading %s / %s: %w", r.certPath, r.keyPath, err)
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.cert = &cert
|
||||
r.certMod, r.keyMod = modTime(r.certPath), modTime(r.keyPath)
|
||||
return nil
|
||||
}
|
||||
|
||||
// NotAfter is when the loaded certificate expires, for the admin UI to show and for a startup
|
||||
// warning. An expiry an operator can see is one they can act on before a browser tells them.
|
||||
func (r *Reloader) NotAfter() time.Time {
|
||||
r.mu.RLock()
|
||||
defer r.mu.RUnlock()
|
||||
if r.cert == nil || r.cert.Leaf == nil {
|
||||
return time.Time{}
|
||||
}
|
||||
return r.cert.Leaf.NotAfter
|
||||
}
|
||||
|
||||
func modTime(path string) time.Time {
|
||||
fi, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
return fi.ModTime()
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package certreload
|
||||
|
||||
import (
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/pem"
|
||||
"math/big"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func writePair(t *testing.T, dir, cn string) (string, string) {
|
||||
t.Helper()
|
||||
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tmpl := x509.Certificate{
|
||||
SerialNumber: big.NewInt(time.Now().UnixNano()),
|
||||
Subject: pkix.Name{CommonName: cn},
|
||||
NotBefore: time.Now().Add(-time.Hour),
|
||||
NotAfter: time.Now().Add(24 * time.Hour),
|
||||
}
|
||||
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &key.PublicKey, key)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
certPath := filepath.Join(dir, "cert.pem")
|
||||
keyPath := filepath.Join(dir, "key.pem")
|
||||
cb := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
kb, _ := x509.MarshalECPrivateKey(key)
|
||||
if err := os.WriteFile(certPath, cb, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: kb}), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return certPath, keyPath
|
||||
}
|
||||
|
||||
func TestBadPathsFailAtStartupNotAtHandshake(t *testing.T) {
|
||||
if _, err := New("/nonexistent/cert.pem", "/nonexistent/key.pem"); err == nil {
|
||||
t.Fatal("a missing certificate was accepted; the failure would surface as an " +
|
||||
"unexplained connection error at the client instead")
|
||||
}
|
||||
}
|
||||
|
||||
// The whole point: a renewal that drops new files in place is picked up without a restart and
|
||||
// without a reload hook that can silently stop working.
|
||||
func TestANewCertificateOnDiskIsPickedUp(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
certPath, keyPath := writePair(t, dir, "first")
|
||||
r, err := New(certPath, keyPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.interval = 0 // check on every handshake, rather than waiting out the rate limit
|
||||
|
||||
got, err := r.getCertificate(nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first := got.Leaf
|
||||
|
||||
time.Sleep(10 * time.Millisecond) // ensure a distinct mtime
|
||||
writePair(t, dir, "second")
|
||||
|
||||
got, err = r.getCertificate(nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Leaf != nil && first != nil && got.Leaf.SerialNumber.Cmp(first.SerialNumber) == 0 {
|
||||
t.Fatal("the replaced certificate was not picked up")
|
||||
}
|
||||
}
|
||||
|
||||
// Renewal tools write the certificate and the key separately, so there is a window where the two
|
||||
// do not match. Serving the previous certificate through it beats serving none.
|
||||
func TestAHalfWrittenPairKeepsTheOldCertificate(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
certPath, keyPath := writePair(t, dir, "good")
|
||||
r, err := New(certPath, keyPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.interval = 0
|
||||
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
if err := os.WriteFile(certPath, []byte("-----BEGIN CERTIFICATE-----\ntruncated\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := r.getCertificate(nil)
|
||||
if err != nil {
|
||||
t.Fatalf("a torn write took the listener down: %v", err)
|
||||
}
|
||||
if got == nil {
|
||||
t.Fatal("no certificate served during a torn write")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExpiryIsVisible(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
certPath, keyPath := writePair(t, dir, "x")
|
||||
r, err := New(certPath, keyPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := r.NotAfter(); got.IsZero() || time.Until(got) > 48*time.Hour {
|
||||
t.Fatalf("expiry not reported sensibly: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,8 @@ package config
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -67,10 +69,57 @@ type Config struct {
|
||||
// first control listen address.
|
||||
PublicControlURL string // ECHOLOT_PUBLIC_URL / --public-url
|
||||
|
||||
// Identity provider. Empty issuer disables sign-in entirely; the server is a relying
|
||||
// party and never stores passwords.
|
||||
OIDCIssuer string // ECHOLOT_OIDC_ISSUER / --oidc-issuer
|
||||
OIDCClientID string // ECHOLOT_OIDC_CLIENT_ID / --oidc-client-id (confidential, admin UI)
|
||||
OIDCAppClientID string // ECHOLOT_OIDC_APP_CLIENT_ID / --oidc-app-client-id (public, the phone app)
|
||||
OIDCAdminGroup string // ECHOLOT_OIDC_ADMIN_GROUP / --oidc-admin-group
|
||||
|
||||
// Break-glass admin username; the password lives hashed in the state store.
|
||||
AdminUser string // ECHOLOT_ADMIN_USER / --admin-user
|
||||
|
||||
// Secret for the *confidential* admin client. Prefer ECHOLOT_OIDC_CLIENT_SECRET_FILE: a path
|
||||
// keeps the secret out of the environment, where it is readable by anything that can see
|
||||
// /proc/<pid>/environ and lands in every dump of the unit's configuration.
|
||||
OIDCClientSecret string // ECHOLOT_OIDC_CLIENT_SECRET / _FILE
|
||||
|
||||
// Where the admin UI is reachable, used to build the OIDC redirect URI. Must match what is
|
||||
// registered at the IdP exactly.
|
||||
AdminBaseURL string // ECHOLOT_ADMIN_BASE_URL / --admin-base-url
|
||||
// TLS for the admin listener. Without these it serves plaintext, which is only acceptable on
|
||||
// loopback — see checkAdminExposure.
|
||||
AdminTLSCert string // ECHOLOT_ADMIN_TLS_CERT / --admin-tls-cert
|
||||
AdminTLSKey string // ECHOLOT_ADMIN_TLS_KEY / --admin-tls-key
|
||||
// Deliberate override for serving the admin UI in plaintext off loopback, so that decision
|
||||
// is made rather than stumbled into.
|
||||
AdminInsecure bool // ECHOLOT_ADMIN_INSECURE / --admin-insecure
|
||||
|
||||
// Port-80 listener that answers ACME HTTP-01 challenges and redirects everything else to
|
||||
// the admin UI. Empty disables it. HTTP-01 always arrives on port 80 — the CA picks the
|
||||
// port — so this never collides with the admin UI on 443.
|
||||
ACMEHTTPListen string // ECHOLOT_ACME_HTTP_LISTEN / --acme-http-listen
|
||||
// Directory an ACME client writes challenge tokens into. Defaults to <state-dir>/acme.
|
||||
ACMEWebroot string // ECHOLOT_ACME_WEBROOT / --acme-webroot
|
||||
|
||||
// Mode
|
||||
Docker bool // --docker (or autodetected; env ECHOLOT_DOCKER=1 forces)
|
||||
}
|
||||
|
||||
// secretOr reads ECHOLOT_<key>, or the contents of the file named by ECHOLOT_<key>_FILE.
|
||||
//
|
||||
// The file form exists because a secret in the environment is readable by anything that can see
|
||||
// /proc/<pid>/environ and lands in every dump of the unit's configuration. A path costs nothing
|
||||
// and keeps the value in one file whose permissions an operator can reason about.
|
||||
func secretOr(key, def string) string {
|
||||
if path := envOr(key+"_FILE", ""); path != "" {
|
||||
if b, err := os.ReadFile(path); err == nil {
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
}
|
||||
return envOr(key, def)
|
||||
}
|
||||
|
||||
// envInt reads ECHOLOT_<key> as an integer with a fallback.
|
||||
func envInt(key string, def int) int {
|
||||
if v := envOr(key, ""); v != "" {
|
||||
@@ -115,6 +164,17 @@ func Load(args []string) (*Config, *Actions, error) {
|
||||
fs.IntVar(&c.UploadRetentionDays, "upload-retention-days", envInt("UPLOAD_RETENTION_DAYS", 90), "delete uploaded runs older than this; 0 disables")
|
||||
fs.IntVar(&c.UploadMaxRuns, "upload-max-runs", envInt("UPLOAD_MAX_RUNS", 200), "keep at most this many runs per device; 0 disables")
|
||||
fs.StringVar(&c.UploadMinAnon, "upload-min-anonymization", envOr("UPLOAD_MIN_ANONYMIZATION", "full"), "least anonymization accepted: full|balanced|strict")
|
||||
fs.StringVar(&c.OIDCIssuer, "oidc-issuer", envOr("OIDC_ISSUER", ""), "OpenID Connect issuer URL; empty disables sign-in")
|
||||
fs.StringVar(&c.OIDCClientID, "oidc-client-id", envOr("OIDC_CLIENT_ID", ""), "confidential OIDC client id for the admin UI")
|
||||
fs.StringVar(&c.OIDCAppClientID, "oidc-app-client-id", envOr("OIDC_APP_CLIENT_ID", ""), "public OIDC client id used by the Android app (PKCE)")
|
||||
fs.StringVar(&c.OIDCAdminGroup, "oidc-admin-group", envOr("OIDC_ADMIN_GROUP", ""), "group claim required for admin access; empty means nobody is an admin via OIDC")
|
||||
fs.StringVar(&c.OIDCClientSecret, "oidc-client-secret", secretOr("OIDC_CLIENT_SECRET", ""), "secret for the confidential admin client; prefer ECHOLOT_OIDC_CLIENT_SECRET_FILE")
|
||||
fs.StringVar(&c.AdminBaseURL, "admin-base-url", envOr("ADMIN_BASE_URL", ""), "public URL of the admin UI, for the OIDC redirect (e.g. https://admin.example.net)")
|
||||
fs.StringVar(&c.AdminTLSCert, "admin-tls-cert", envOr("ADMIN_TLS_CERT", ""), "TLS certificate for the admin listener")
|
||||
fs.StringVar(&c.AdminTLSKey, "admin-tls-key", envOr("ADMIN_TLS_KEY", ""), "TLS key for the admin listener")
|
||||
fs.BoolVar(&c.AdminInsecure, "admin-insecure", envOr("ADMIN_INSECURE", "") == "1", "allow the admin UI in plaintext off loopback (you are on your own)")
|
||||
fs.StringVar(&c.ACMEHTTPListen, "acme-http-listen", envOr("ACME_HTTP_LISTEN", ""), "port-80 listener for ACME HTTP-01 challenges and http->https redirects")
|
||||
fs.StringVar(&c.ACMEWebroot, "acme-webroot", envOr("ACME_WEBROOT", ""), "directory an ACME client writes challenges into (default <state-dir>/acme)")
|
||||
fs.StringVar(&c.PublicControlURL, "public-url", envOr("PUBLIC_URL", ""), "public control-plane URL for enrollment links, e.g. https://probe.example.net:8443")
|
||||
fs.StringVar(&c.MinAppVersion, "min-app-version", envOr("MIN_APP_VERSION", "0.2.0"), "oldest app version this server will serve (SemVer, inclusive)")
|
||||
fs.StringVar(&c.MaxAppVersion, "max-app-version", envOr("MAX_APP_VERSION", "1.0.0"), "first app version this server will refuse (SemVer, exclusive); empty = unbounded")
|
||||
@@ -122,6 +182,12 @@ func Load(args []string) (*Config, *Actions, error) {
|
||||
|
||||
fs.BoolVar(&a.InstallSystemd, "install-systemd", false, "install a systemd unit for this binary and exit")
|
||||
fs.BoolVar(&a.UninstallSystemd, "uninstall-systemd", false, "remove the systemd unit and exit")
|
||||
var daemon bool
|
||||
fs.BoolVar(&a.Serve, "serve", false, "run the server (bind listeners and answer requests)")
|
||||
fs.BoolVar(&daemon, "daemon", false, "alias for --serve")
|
||||
fs.BoolVar(&a.SetAdminPassword, "set-admin-password", false,
|
||||
"set the break-glass admin password (username as --admin-user, password read from stdin) and exit")
|
||||
fs.StringVar(&c.AdminUser, "admin-user", envOr("ADMIN_USER", "admin"), "username for the break-glass admin")
|
||||
fs.BoolVar(&a.SelfUpdate, "self-update", false, "check for a newer release, replace this binary, and exit")
|
||||
fs.BoolVar(&a.Version, "version", false, "print version and exit")
|
||||
|
||||
@@ -131,12 +197,78 @@ func Load(args []string) (*Config, *Actions, error) {
|
||||
if !c.Docker {
|
||||
c.Docker = inContainer()
|
||||
}
|
||||
a.Serve = a.Serve || daemon
|
||||
// No verb at all means the caller has not said what they want. Usage is the answer, and it
|
||||
// is a usage error rather than success — otherwise a service manager sees a clean exit and
|
||||
// concludes the server ran and finished.
|
||||
if !a.Serve && !a.InstallSystemd && !a.UninstallSystemd && !a.SelfUpdate &&
|
||||
!a.SetAdminPassword && !a.Version {
|
||||
a.Help = true
|
||||
}
|
||||
if a.Serve {
|
||||
if err := c.checkAdminExposure(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
if c.Docker && (a.InstallSystemd || a.UninstallSystemd || a.SelfUpdate) {
|
||||
return nil, nil, fmt.Errorf("systemd/self-update actions are native-mode only (container detected; override with ECHOLOT_DOCKER=0 if this is wrong)")
|
||||
}
|
||||
return c, a, nil
|
||||
}
|
||||
|
||||
// Usage prints the verbs first and the tuning flags second, because the question someone has
|
||||
// when they run this by name is "what does it do", not "what can I set".
|
||||
func Usage(w io.Writer) {
|
||||
fmt.Fprint(w, `echolot-server — the Echolot probe server
|
||||
|
||||
USAGE
|
||||
echolot-server --serve run the server
|
||||
echolot-server --version print the version
|
||||
echolot-server --install-systemd install and enable a systemd unit
|
||||
echolot-server --uninstall-systemd remove it
|
||||
echolot-server --self-update replace this binary with the latest release
|
||||
echolot-server --set-admin-password set the break-glass admin password (stdin)
|
||||
echolot-server --help full flag list
|
||||
|
||||
Every flag can also be set as an environment variable: --control-listen becomes
|
||||
ECHOLOT_CONTROL_LISTEN. In a container, configuration comes from the environment.
|
||||
|
||||
Running with no verb prints this and exits non-zero: starting to serve the internet
|
||||
should be something you asked for.
|
||||
`)
|
||||
}
|
||||
|
||||
// checkAdminExposure refuses to serve an unencrypted admin UI on a non-loopback address.
|
||||
//
|
||||
// The admin session cookie is a bearer credential for everything this server can do, and the OIDC
|
||||
// authorization code arrives in a URL. In plaintext, both are readable by anyone on the path — and
|
||||
// on a globally routable address "the path" means the internet. This is a hard stop rather than a
|
||||
// warning because a warning in a log is not read by the person who most needs it, and because the
|
||||
// two safe answers are cheap: bind to loopback and tunnel, or supply a certificate.
|
||||
func (c *Config) checkAdminExposure() error {
|
||||
if c.AdminTLSCert != "" || c.AdminInsecure {
|
||||
return nil
|
||||
}
|
||||
for _, addr := range Addrs(c.AdminListen) {
|
||||
host, _, err := net.SplitHostPort(addr)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ip := net.ParseIP(strings.Trim(host, "[]"))
|
||||
if host == "" || ip == nil || ip.IsLoopback() {
|
||||
continue // loopback, or a name we cannot judge; RFC 8252 blesses loopback plaintext
|
||||
}
|
||||
return fmt.Errorf(
|
||||
"refusing to serve the admin UI in plaintext on %s: the session cookie and the OIDC "+
|
||||
"authorization code would cross the network in the clear.\n"+
|
||||
" Fix it one of three ways:\n"+
|
||||
" - bind to 127.0.0.1 and reach it over an SSH tunnel (no certificate needed)\n"+
|
||||
" - set ECHOLOT_ADMIN_TLS_CERT and ECHOLOT_ADMIN_TLS_KEY\n"+
|
||||
" - set ECHOLOT_ADMIN_INSECURE=1 if you genuinely mean it", addr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Addrs splits a comma-separated listen spec into individual addresses.
|
||||
// Explicit per-address binds matter on multi-IP hosts: a wildcard bind
|
||||
// (":8443") would also claim addresses reserved for other purposes (e.g. an
|
||||
@@ -151,11 +283,19 @@ func Addrs(spec string) []string {
|
||||
return out
|
||||
}
|
||||
|
||||
// Actions are one-shot verbs that exit instead of serving.
|
||||
// Actions are the verbs. Serving is one of them, and it is explicit: running the binary with no
|
||||
// arguments prints usage rather than binding a dozen ports and starting to answer the internet.
|
||||
// Someone typing the name of an unfamiliar program on a terminal should be told what it does, not
|
||||
// have it start doing it.
|
||||
type Actions struct {
|
||||
Serve bool
|
||||
// Help is set when there is nothing to do: no verb was given.
|
||||
Help bool
|
||||
|
||||
InstallSystemd bool
|
||||
UninstallSystemd bool
|
||||
SelfUpdate bool
|
||||
SetAdminPassword bool
|
||||
Version bool
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
@@ -27,6 +28,7 @@ import (
|
||||
|
||||
"echo-lot.app/server/internal/compat"
|
||||
"echo-lot.app/server/internal/dataplane"
|
||||
"echo-lot.app/server/internal/oidc"
|
||||
"echo-lot.app/server/internal/runs"
|
||||
"echo-lot.app/server/internal/session"
|
||||
"echo-lot.app/server/internal/store"
|
||||
@@ -57,6 +59,8 @@ type Server struct {
|
||||
// Granted server->client sends (spec §5). Both consume an asymmetric grant.
|
||||
DownTrain func(sess *session.Session, g *session.Grant, count, sizeBytes, intervalUs int) (int, error)
|
||||
BigSend func(sess *session.Session, g *session.Grant, sizes []int, df bool) ([]dataplane.BigSendResult, error)
|
||||
// OIDC verifies ID tokens when the operator has configured an issuer (may be nil).
|
||||
OIDC *oidc.Verifier
|
||||
// Runs stores uploaded measurement documents (may be nil: uploads unsupported).
|
||||
Runs *runs.Store
|
||||
// FragSend emits one datagram as hand-built IP fragments in a chosen order (may be nil:
|
||||
@@ -162,6 +166,9 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /v1/runs", gate(s.listRuns))
|
||||
mux.HandleFunc("GET /v1/runs/{id}", gate(s.getRun))
|
||||
mux.HandleFunc("DELETE /v1/runs/{id}", gate(s.deleteRun))
|
||||
mux.HandleFunc("POST /v1/account/link", gate(s.linkAccount))
|
||||
mux.HandleFunc("DELETE /v1/account/link", gate(s.unlinkAccount))
|
||||
mux.HandleFunc("GET /v1/account", gate(s.accountStatus))
|
||||
// TODO(spec §5): frag_send, throughput (both build on the same grant machinery)
|
||||
return mux
|
||||
}
|
||||
@@ -225,6 +232,8 @@ func (s *Server) observations(w http.ResponseWriter, r *http.Request) {
|
||||
"connect_back": cb,
|
||||
// The sender's own count, which is what makes the receiver's count mean something.
|
||||
"throughput": sess.ThroughputReports(),
|
||||
// The receiver's count for upstream runs — same idea, other direction.
|
||||
"throughput_up": upstreamJSON(sess),
|
||||
"dns_canary": dnsCanary,
|
||||
// TODO(spec §6): http echo records
|
||||
})
|
||||
@@ -433,9 +442,20 @@ func (s *Server) actions(w http.ResponseWriter, r *http.Request) {
|
||||
// Only the downstream direction needs the server to send. Upstream is the client
|
||||
// sending and the server counting, which needs no action at all — so asking for it here
|
||||
// is a client bug worth naming rather than silently doing the other thing.
|
||||
if req.Direction == "up" {
|
||||
// Upstream needs nothing sent from here — the client generates the traffic and the
|
||||
// server counts it. The only thing an action can usefully do is zero the counter so
|
||||
// the run measures itself rather than inheriting an earlier one.
|
||||
sess.ResetUpstream()
|
||||
writeJSON(w, http.StatusAccepted, map[string]any{
|
||||
"action_id": actionID, "direction": "up", "reset": true,
|
||||
"note": "send TYPE_THROUGHPUT_UP packets, then read observations.throughput_up",
|
||||
})
|
||||
return
|
||||
}
|
||||
if req.Direction != "" && req.Direction != "down" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{
|
||||
"error": "only direction=down is an action; upstream throughput is measured by sending and reading observations",
|
||||
"error": "direction must be up or down",
|
||||
})
|
||||
return
|
||||
}
|
||||
@@ -605,6 +625,10 @@ func (s *Server) profile(w http.ResponseWriter, r *http.Request) {
|
||||
// The app needs the upload rules before it offers the switch: whether uploads are
|
||||
// accepted at all, and how much identifying detail it must strip first.
|
||||
"uploads": s.uploadPolicy(),
|
||||
// What a client needs to start a sign-in, without hard-coding the operator's IdP into
|
||||
// the app: where to authorize, which client id to use, and whether it is worth offering
|
||||
// sign-in at all on this server.
|
||||
"auth": s.authInfo(r.Context()),
|
||||
// What this build speaks, and which app versions it will serve. A client checks the
|
||||
// server side of the same question against its own bounds.
|
||||
"compat": map[string]any{
|
||||
@@ -701,7 +725,7 @@ func (s *Server) uploadRun(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "read failed"})
|
||||
return
|
||||
}
|
||||
meta, err := s.Runs.Put(dev.ID, body)
|
||||
meta, err := s.Runs.Put(dev.ID, body, dev.LinkedToAccount())
|
||||
switch {
|
||||
case err == nil:
|
||||
slog.Info("run uploaded", "device", dev.ID, "run", meta.ID,
|
||||
@@ -785,3 +809,118 @@ func (s *Server) EnrollmentLink(token string) string {
|
||||
"&p=" + url.QueryEscape("pin-sha256:"+s.PinB64) +
|
||||
"&t=" + url.QueryEscape(token)
|
||||
}
|
||||
|
||||
// upstreamJSON renders the upstream tally with the derived figures already computed, so every
|
||||
// consumer does not have to repeat (and risk fumbling) the same arithmetic.
|
||||
func upstreamJSON(sess *session.Session) map[string]any {
|
||||
u := sess.Upstream()
|
||||
return map[string]any{
|
||||
"packets": u.Packets, "bytes": u.Bytes,
|
||||
"span_ms": u.SpanMs(), "kbps": u.Kbps(),
|
||||
}
|
||||
}
|
||||
|
||||
// authInfo advertises the sign-in configuration, so the app can present a Sign in button only
|
||||
// when there is something behind it, and can drive the flow without the user typing an issuer URL.
|
||||
func (s *Server) authInfo(ctx context.Context) map[string]any {
|
||||
if s.OIDC == nil || !s.OIDC.Config().Enabled() {
|
||||
return map[string]any{"enabled": false}
|
||||
}
|
||||
cfg := s.OIDC.Config()
|
||||
out := map[string]any{
|
||||
"enabled": true,
|
||||
"issuer": cfg.Issuer,
|
||||
// The app's client, not the server's: this is what a phone should authorize as.
|
||||
"client_id": cfg.AppClientID,
|
||||
// The app is a public client on a phone: no secret can be kept, so PKCE is what
|
||||
// protects the code exchange (RFC 7636), and the redirect comes back through the
|
||||
// scheme the app already registers for enrollment links.
|
||||
"flow": "authorization_code+pkce",
|
||||
"redirect_uri": "echolot://auth",
|
||||
"scopes": "openid profile email",
|
||||
}
|
||||
if d, err := s.OIDC.Discover(ctx); err == nil {
|
||||
out["authorization_endpoint"] = d.AuthorizationEndpoint
|
||||
out["token_endpoint"] = d.TokenEndpoint
|
||||
out["end_session_endpoint"] = d.EndSessionEndpoint
|
||||
} else {
|
||||
// Reported rather than hidden: an unreachable IdP is the operator's problem to see, and
|
||||
// a client that knows the difference can say "sign-in is configured but the provider is
|
||||
// not answering" instead of failing obscurely.
|
||||
out["discovery_error"] = err.Error()
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// linkAccount ties the calling device to the person whose ID token it presents.
|
||||
//
|
||||
// The device credential proves *which device*; the ID token proves *which person*. Both are
|
||||
// required, and neither substitutes for the other: enrollment admits a device to the server,
|
||||
// signing in attributes it to someone.
|
||||
func (s *Server) linkAccount(w http.ResponseWriter, r *http.Request) {
|
||||
dev := s.Store.DeviceByCredential(bearer(r))
|
||||
if dev == nil {
|
||||
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "unknown credential"})
|
||||
return
|
||||
}
|
||||
if s.OIDC == nil || !s.OIDC.Config().Enabled() {
|
||||
writeJSON(w, http.StatusNotImplemented, map[string]string{
|
||||
"error": "this server has no identity provider configured, so there is nothing to sign in to",
|
||||
})
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
IDToken string `json:"id_token"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || body.IDToken == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "expected an id_token"})
|
||||
return
|
||||
}
|
||||
claims, err := s.OIDC.Verify(r.Context(), body.IDToken)
|
||||
if err != nil {
|
||||
// Deliberately terse to the client and detailed in the log: a caller probing token
|
||||
// handling should not be told which check it failed.
|
||||
slog.Info("rejected sign-in", "device", dev.ID, "err", err)
|
||||
writeJSON(w, http.StatusForbidden, map[string]string{"error": "the identity token was not accepted"})
|
||||
return
|
||||
}
|
||||
if err := s.Store.LinkAccount(dev.ID, claims.AccountID(), claims.Display()); err != nil {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
slog.Info("device linked to account", "device", dev.ID, "account", claims.AccountID(),
|
||||
"admin", s.OIDC.IsAdmin(claims))
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"account_id": claims.AccountID(), "display_name": claims.Display(),
|
||||
"admin": s.OIDC.IsAdmin(claims),
|
||||
})
|
||||
}
|
||||
|
||||
// unlinkAccount signs out on this device. The device stays enrolled: signing out should not cost
|
||||
// someone their enrollment, which an operator had to grant.
|
||||
func (s *Server) unlinkAccount(w http.ResponseWriter, r *http.Request) {
|
||||
dev := s.Store.DeviceByCredential(bearer(r))
|
||||
if dev == nil {
|
||||
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "unknown credential"})
|
||||
return
|
||||
}
|
||||
if err := s.Store.LinkAccount(dev.ID, "", ""); err != nil {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (s *Server) accountStatus(w http.ResponseWriter, r *http.Request) {
|
||||
dev := s.Store.DeviceByCredential(bearer(r))
|
||||
if dev == nil {
|
||||
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "unknown credential"})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"signed_in": dev.LinkedToAccount(),
|
||||
"account_id": dev.AccountID,
|
||||
"display_name": dev.AccountName,
|
||||
"device_id": dev.ID,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -39,6 +39,10 @@ const (
|
||||
TypeFragData = 0x0D
|
||||
// TypeThroughputData is one packet of a sustained-rate downstream run.
|
||||
TypeThroughputData = 0x0E
|
||||
// TypeThroughputUp is one packet of a client-driven upstream run. The server counts it and
|
||||
// deliberately does not answer: a reply would double the traffic and measure the return
|
||||
// path at the same time, which is the one thing this test is trying not to do.
|
||||
TypeThroughputUp = 0x0F
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
@@ -152,6 +156,14 @@ func (s *Server) handle(conn *net.UDPConn, raddr netip.AddrPort, pkt []byte, tRx
|
||||
if la, ok := conn.LocalAddr().(*net.UDPAddr); ok {
|
||||
sess.NoteDataLocal(la.AddrPort())
|
||||
}
|
||||
|
||||
// Upstream throughput short-circuits before the observation log. Recording one struct per
|
||||
// packet here would mean tens of thousands of allocations for a single run; the counter is
|
||||
// all anyone needs, since the client holds the send-side record.
|
||||
if typ == TypeThroughputUp {
|
||||
sess.CountUpstream(len(pkt), tRxNs)
|
||||
return
|
||||
}
|
||||
sess.RecordUDP(session.UDPObservation{
|
||||
Seq: seq, TRxNs: tRxNs, TTxNs: time.Since(s.start).Nanoseconds(),
|
||||
Src: raddr.String(), Size: len(pkt), Type: typ,
|
||||
|
||||
@@ -0,0 +1,504 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
// Package oidc verifies OpenID Connect ID tokens against a configured issuer.
|
||||
//
|
||||
// Echolot is a *relying party*, never an identity provider. It delegates to whatever IdP the
|
||||
// operator already runs and stores no passwords — no hashing, no reset flow, no lockout policy,
|
||||
// and no credential database to leak. For a tool people self-host on a box they also use for
|
||||
// other things, that is the difference between "one more service" and "one more thing that can
|
||||
// lose your users' passwords".
|
||||
//
|
||||
// Verification is written against the stdlib rather than a JWT library, because the server has no
|
||||
// external dependencies by design. That is a real constraint and it cuts both ways: the code below
|
||||
// is longer than `jwt.Parse`, but it is also auditable in one sitting and cannot be broken by
|
||||
// somebody else's release. The algorithm allow-list is the part that matters — accepting `alg`
|
||||
// from the token itself is the classic JWT forgery, so it is fixed here and `none` can never
|
||||
// appear.
|
||||
package oidc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rsa"
|
||||
"crypto/sha256"
|
||||
"crypto/sha512"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Claims are the parts of an ID token Echolot acts on.
|
||||
type Claims struct {
|
||||
Issuer string `json:"iss"`
|
||||
Subject string `json:"sub"`
|
||||
Audience audience `json:"aud"`
|
||||
Expiry int64 `json:"exp"`
|
||||
IssuedAt int64 `json:"iat"`
|
||||
Nonce string `json:"nonce"`
|
||||
Email string `json:"email"`
|
||||
Name string `json:"name"`
|
||||
Username string `json:"preferred_username"`
|
||||
Groups []string `json:"groups"`
|
||||
}
|
||||
|
||||
// AccountID is the stable identity of a person: issuer plus subject.
|
||||
//
|
||||
// Subject alone is not enough — it is only unique within an issuer — and email is not stable,
|
||||
// since people change them and IdPs allow reuse. Keying on iss+sub means an operator can switch
|
||||
// IdPs and know that the accounts did not silently merge.
|
||||
func (c Claims) AccountID() string { return c.Issuer + "#" + c.Subject }
|
||||
|
||||
// Display is the friendliest name available, for the admin UI.
|
||||
func (c Claims) Display() string {
|
||||
for _, s := range []string{c.Name, c.Username, c.Email} {
|
||||
if s != "" {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return c.Subject
|
||||
}
|
||||
|
||||
// audience tolerates the spec's two shapes: a string or an array of strings.
|
||||
type audience []string
|
||||
|
||||
func (a *audience) UnmarshalJSON(b []byte) error {
|
||||
var one string
|
||||
if err := json.Unmarshal(b, &one); err == nil {
|
||||
*a = audience{one}
|
||||
return nil
|
||||
}
|
||||
var many []string
|
||||
if err := json.Unmarshal(b, &many); err != nil {
|
||||
return err
|
||||
}
|
||||
*a = many
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a audience) contains(s string) bool {
|
||||
for _, v := range a {
|
||||
if v == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Config is what the operator supplies.
|
||||
type Config struct {
|
||||
// Issuer is the IdP's base URL, e.g. https://auth.example.net/application/o/echolot/
|
||||
Issuer string
|
||||
// ClientID is this server's own registered client — confidential, used for the admin UI's
|
||||
// browser login, where a secret can genuinely be kept in the host's config.
|
||||
ClientID string
|
||||
// AppClientID is the mobile app's registered client. It is a separate, *public* client
|
||||
// because an APK cannot keep a secret, so it uses PKCE instead.
|
||||
//
|
||||
// Both are accepted as audiences, and they must be listed rather than merged: a token is
|
||||
// addressed to a specific client, and accepting "any client of this issuer" would let every
|
||||
// other application registered with the same IdP authenticate here.
|
||||
AppClientID string
|
||||
// AdminGroup, when set, is the group claim a person must hold to reach the admin UI.
|
||||
// Empty means no one is an admin via OIDC, which is the safe default: an operator who has
|
||||
// not said who may administer the server has not said "everyone".
|
||||
AdminGroup string
|
||||
// Skew tolerated on exp/iat, for ordinary clock drift between the IdP and this server.
|
||||
Skew time.Duration
|
||||
}
|
||||
|
||||
func (c Config) Enabled() bool { return c.Issuer != "" && (c.ClientID != "" || c.AppClientID != "") }
|
||||
|
||||
// acceptedAudiences is every client id this server answers for.
|
||||
func (v *Verifier) acceptedAudiences() []string {
|
||||
out := make([]string, 0, 2)
|
||||
for _, id := range []string{v.cfg.ClientID, v.cfg.AppClientID} {
|
||||
if id != "" {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (v *Verifier) audienceAccepted(aud audience) bool {
|
||||
for _, id := range v.acceptedAudiences() {
|
||||
if aud.contains(id) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Discovery is the subset of the provider metadata document that is used.
|
||||
type Discovery struct {
|
||||
Issuer string `json:"issuer"`
|
||||
AuthorizationEndpoint string `json:"authorization_endpoint"`
|
||||
TokenEndpoint string `json:"token_endpoint"`
|
||||
JWKSURI string `json:"jwks_uri"`
|
||||
UserinfoEndpoint string `json:"userinfo_endpoint"`
|
||||
EndSessionEndpoint string `json:"end_session_endpoint"`
|
||||
}
|
||||
|
||||
// Verifier fetches provider metadata and keys, and checks tokens against them.
|
||||
type Verifier struct {
|
||||
cfg Config
|
||||
client *http.Client
|
||||
|
||||
mu sync.RWMutex
|
||||
discovery *Discovery
|
||||
keys map[string]crypto.PublicKey
|
||||
keysAt time.Time
|
||||
}
|
||||
|
||||
func New(cfg Config, client *http.Client) *Verifier {
|
||||
if cfg.Skew == 0 {
|
||||
cfg.Skew = 2 * time.Minute
|
||||
}
|
||||
if client == nil {
|
||||
client = &http.Client{Timeout: 10 * time.Second}
|
||||
}
|
||||
return &Verifier{cfg: cfg, client: client, keys: map[string]crypto.PublicKey{}}
|
||||
}
|
||||
|
||||
func (v *Verifier) Config() Config { return v.cfg }
|
||||
|
||||
var (
|
||||
ErrDisabled = errors.New("no OIDC issuer is configured on this server")
|
||||
ErrMalformed = errors.New("token is not a well-formed JWT")
|
||||
ErrSignature = errors.New("token signature does not verify")
|
||||
ErrClaims = errors.New("token claims are not acceptable")
|
||||
)
|
||||
|
||||
// Discover fetches (and caches) the provider metadata.
|
||||
func (v *Verifier) Discover(ctx context.Context) (*Discovery, error) {
|
||||
if !v.cfg.Enabled() {
|
||||
return nil, ErrDisabled
|
||||
}
|
||||
v.mu.RLock()
|
||||
d := v.discovery
|
||||
v.mu.RUnlock()
|
||||
if d != nil {
|
||||
return d, nil
|
||||
}
|
||||
|
||||
url := strings.TrimRight(v.cfg.Issuer, "/") + "/.well-known/openid-configuration"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := v.client.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("discovery: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("discovery: %s returned %d", url, resp.StatusCode)
|
||||
}
|
||||
var got Discovery
|
||||
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
|
||||
return nil, fmt.Errorf("discovery: %w", err)
|
||||
}
|
||||
// The issuer in the document must match the one configured, or a redirect could point us at
|
||||
// somebody else's keys while we keep believing we are talking to the configured provider.
|
||||
if strings.TrimRight(got.Issuer, "/") != strings.TrimRight(v.cfg.Issuer, "/") {
|
||||
return nil, fmt.Errorf("discovery: document says issuer %q, configured %q", got.Issuer, v.cfg.Issuer)
|
||||
}
|
||||
v.mu.Lock()
|
||||
v.discovery = &got
|
||||
v.mu.Unlock()
|
||||
return &got, nil
|
||||
}
|
||||
|
||||
// jwksTTL is how long keys are trusted before refetching. Short enough to pick up a rotation
|
||||
// without an operator restarting anything; long enough that token checks are not IdP round trips.
|
||||
const jwksTTL = 15 * time.Minute
|
||||
|
||||
func (v *Verifier) keyFor(ctx context.Context, kid string) (crypto.PublicKey, error) {
|
||||
v.mu.RLock()
|
||||
k, ok := v.keys[kid]
|
||||
fresh := time.Since(v.keysAt) < jwksTTL
|
||||
v.mu.RUnlock()
|
||||
if ok && fresh {
|
||||
return k, nil
|
||||
}
|
||||
if err := v.refreshKeys(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v.mu.RLock()
|
||||
defer v.mu.RUnlock()
|
||||
if k, ok := v.keys[kid]; ok {
|
||||
return k, nil
|
||||
}
|
||||
// A kid we have never seen, after a refresh, is a token from somewhere else.
|
||||
return nil, fmt.Errorf("%w: no key %q at the issuer", ErrSignature, kid)
|
||||
}
|
||||
|
||||
func (v *Verifier) refreshKeys(ctx context.Context) error {
|
||||
d, err := v.Discover(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, d.JWKSURI, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := v.client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("jwks: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("jwks: %s returned %d", d.JWKSURI, resp.StatusCode)
|
||||
}
|
||||
var set struct {
|
||||
Keys []jwk `json:"keys"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&set); err != nil {
|
||||
return fmt.Errorf("jwks: %w", err)
|
||||
}
|
||||
parsed := make(map[string]crypto.PublicKey, len(set.Keys))
|
||||
for _, k := range set.Keys {
|
||||
if pub, err := k.publicKey(); err == nil {
|
||||
parsed[k.Kid] = pub
|
||||
}
|
||||
}
|
||||
if len(parsed) == 0 {
|
||||
return errors.New("jwks: no usable keys")
|
||||
}
|
||||
v.mu.Lock()
|
||||
v.keys = parsed
|
||||
v.keysAt = time.Now()
|
||||
v.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
type jwk struct {
|
||||
Kty string `json:"kty"`
|
||||
Kid string `json:"kid"`
|
||||
Alg string `json:"alg"`
|
||||
Use string `json:"use"`
|
||||
N string `json:"n"`
|
||||
E string `json:"e"`
|
||||
Crv string `json:"crv"`
|
||||
X string `json:"x"`
|
||||
Y string `json:"y"`
|
||||
}
|
||||
|
||||
func (k jwk) publicKey() (crypto.PublicKey, error) {
|
||||
switch k.Kty {
|
||||
case "RSA":
|
||||
n, err := b64uint(k.N)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
e, err := b64uint(k.E)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !e.IsInt64() || e.Int64() > 1<<31 {
|
||||
return nil, errors.New("implausible RSA exponent")
|
||||
}
|
||||
return &rsa.PublicKey{N: n, E: int(e.Int64())}, nil
|
||||
case "EC":
|
||||
curve, err := curveFor(k.Crv)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
x, err := b64uint(k.X)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
y, err := b64uint(k.Y)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ecdsa.PublicKey{Curve: curve, X: x, Y: y}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("unsupported key type %q", k.Kty)
|
||||
}
|
||||
|
||||
// Verify checks a serialized ID token and returns its claims.
|
||||
//
|
||||
// The order is deliberate: structure, then algorithm, then signature, then claims. Nothing about
|
||||
// the token's contents is believed before its signature has been checked — reading `iss` or `aud`
|
||||
// out of an unverified token and acting on it is how "verified" tokens turn out not to be.
|
||||
func (v *Verifier) Verify(ctx context.Context, token string) (*Claims, error) {
|
||||
if !v.cfg.Enabled() {
|
||||
return nil, ErrDisabled
|
||||
}
|
||||
parts := strings.Split(token, ".")
|
||||
if len(parts) != 3 {
|
||||
return nil, ErrMalformed
|
||||
}
|
||||
headerJSON, err := b64(parts[0])
|
||||
if err != nil {
|
||||
return nil, ErrMalformed
|
||||
}
|
||||
var hdr struct {
|
||||
Alg string `json:"alg"`
|
||||
Kid string `json:"kid"`
|
||||
Typ string `json:"typ"`
|
||||
}
|
||||
if err := json.Unmarshal(headerJSON, &hdr); err != nil {
|
||||
return nil, ErrMalformed
|
||||
}
|
||||
// The allow-list is fixed here rather than taken from the token. Trusting the token's own
|
||||
// `alg` is the classic JWT forgery: "none" turns any token into a valid one, and swapping RS256
|
||||
// for HS256 lets an attacker sign with the public key. Neither is reachable from here.
|
||||
if _, ok := allowedAlgs[hdr.Alg]; !ok {
|
||||
return nil, fmt.Errorf("%w: algorithm %q is not accepted", ErrSignature, hdr.Alg)
|
||||
}
|
||||
|
||||
pub, err := v.keyFor(ctx, hdr.Kid)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sig, err := b64(parts[2])
|
||||
if err != nil {
|
||||
return nil, ErrMalformed
|
||||
}
|
||||
signed := parts[0] + "." + parts[1]
|
||||
if err := verifySignature(hdr.Alg, pub, []byte(signed), sig); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
payload, err := b64(parts[1])
|
||||
if err != nil {
|
||||
return nil, ErrMalformed
|
||||
}
|
||||
var claims Claims
|
||||
if err := json.Unmarshal(payload, &claims); err != nil {
|
||||
return nil, ErrMalformed
|
||||
}
|
||||
if err := v.checkClaims(claims); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &claims, nil
|
||||
}
|
||||
|
||||
func (v *Verifier) checkClaims(c Claims) error {
|
||||
if strings.TrimRight(c.Issuer, "/") != strings.TrimRight(v.cfg.Issuer, "/") {
|
||||
return fmt.Errorf("%w: issued by %q, expected %q", ErrClaims, c.Issuer, v.cfg.Issuer)
|
||||
}
|
||||
// A token addressed to a different client is a valid token that was not meant for us —
|
||||
// accepting it lets any other client of the same IdP authenticate here.
|
||||
if !v.audienceAccepted(c.Audience) {
|
||||
return fmt.Errorf("%w: addressed to %v, not to %v", ErrClaims,
|
||||
[]string(c.Audience), v.acceptedAudiences())
|
||||
}
|
||||
if c.Subject == "" {
|
||||
return fmt.Errorf("%w: no subject", ErrClaims)
|
||||
}
|
||||
now := time.Now()
|
||||
if c.Expiry == 0 || now.After(time.Unix(c.Expiry, 0).Add(v.cfg.Skew)) {
|
||||
return fmt.Errorf("%w: expired", ErrClaims)
|
||||
}
|
||||
if c.IssuedAt != 0 && now.Add(v.cfg.Skew).Before(time.Unix(c.IssuedAt, 0)) {
|
||||
return fmt.Errorf("%w: issued in the future", ErrClaims)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsAdmin reports whether these claims carry the configured admin group.
|
||||
//
|
||||
// With no group configured nobody is an admin: an operator who has not said who may administer
|
||||
// the server has not thereby said "anyone who can log in".
|
||||
func (v *Verifier) IsAdmin(c *Claims) bool {
|
||||
if c == nil || v.cfg.AdminGroup == "" {
|
||||
return false
|
||||
}
|
||||
for _, g := range c.Groups {
|
||||
if g == v.cfg.AdminGroup {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
var allowedAlgs = map[string]crypto.Hash{
|
||||
"RS256": crypto.SHA256, "RS384": crypto.SHA384, "RS512": crypto.SHA512,
|
||||
"ES256": crypto.SHA256, "ES384": crypto.SHA384, "ES512": crypto.SHA512,
|
||||
}
|
||||
|
||||
func verifySignature(alg string, pub crypto.PublicKey, signed, sig []byte) error {
|
||||
h := allowedAlgs[alg]
|
||||
digest := hashOf(h, signed)
|
||||
|
||||
switch {
|
||||
case strings.HasPrefix(alg, "RS"):
|
||||
k, ok := pub.(*rsa.PublicKey)
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: %s token against a non-RSA key", ErrSignature, alg)
|
||||
}
|
||||
if err := rsa.VerifyPKCS1v15(k, h, digest, sig); err != nil {
|
||||
return ErrSignature
|
||||
}
|
||||
return nil
|
||||
case strings.HasPrefix(alg, "ES"):
|
||||
k, ok := pub.(*ecdsa.PublicKey)
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: %s token against a non-EC key", ErrSignature, alg)
|
||||
}
|
||||
// JWS packs ECDSA signatures as r||s, fixed width — not the ASN.1 form ecdsa.Verify
|
||||
// would otherwise expect.
|
||||
if len(sig)%2 != 0 {
|
||||
return ErrSignature
|
||||
}
|
||||
half := len(sig) / 2
|
||||
r := new(big.Int).SetBytes(sig[:half])
|
||||
s := new(big.Int).SetBytes(sig[half:])
|
||||
if !ecdsa.Verify(k, digest, r, s) {
|
||||
return ErrSignature
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return ErrSignature
|
||||
}
|
||||
|
||||
func hashOf(h crypto.Hash, b []byte) []byte {
|
||||
switch h {
|
||||
case crypto.SHA384:
|
||||
d := sha512.Sum384(b)
|
||||
return d[:]
|
||||
case crypto.SHA512:
|
||||
d := sha512.Sum512(b)
|
||||
return d[:]
|
||||
default:
|
||||
d := sha256.Sum256(b)
|
||||
return d[:]
|
||||
}
|
||||
}
|
||||
|
||||
func curveFor(crv string) (elliptic.Curve, error) {
|
||||
switch crv {
|
||||
case "P-256":
|
||||
return elliptic.P256(), nil
|
||||
case "P-384":
|
||||
return elliptic.P384(), nil
|
||||
case "P-521":
|
||||
return elliptic.P521(), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unsupported curve %q", crv)
|
||||
}
|
||||
|
||||
// b64 decodes JWT base64url, which omits padding.
|
||||
func b64(s string) ([]byte, error) { return base64.RawURLEncoding.DecodeString(s) }
|
||||
|
||||
func b64uint(s string) (*big.Int, error) {
|
||||
b, err := b64(s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(b) == 0 {
|
||||
return nil, errors.New("empty value")
|
||||
}
|
||||
return new(big.Int).SetBytes(b), nil
|
||||
}
|
||||
@@ -0,0 +1,325 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package oidc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/rsa"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A self-contained IdP: real keys, real signatures, real discovery and JWKS documents. Testing
|
||||
// token verification against anything less than a genuine signer proves nothing — the failure
|
||||
// modes that matter here (accepting `none`, accepting another client's token, accepting an
|
||||
// expired one) all look fine to a mock that just returns success.
|
||||
type testIdP struct {
|
||||
*httptest.Server
|
||||
rsaKey *rsa.PrivateKey
|
||||
ecKey *ecdsa.PrivateKey
|
||||
}
|
||||
|
||||
func newIdP(t *testing.T) *testIdP {
|
||||
t.Helper()
|
||||
rk, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ek, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
idp := &testIdP{rsaKey: rk, ecKey: ek}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, r *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(Discovery{
|
||||
Issuer: idp.URL,
|
||||
AuthorizationEndpoint: idp.URL + "/auth",
|
||||
TokenEndpoint: idp.URL + "/token",
|
||||
JWKSURI: idp.URL + "/jwks",
|
||||
})
|
||||
})
|
||||
mux.HandleFunc("/jwks", func(w http.ResponseWriter, r *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"keys": []map[string]string{
|
||||
{
|
||||
"kty": "RSA", "kid": "rsa-1", "alg": "RS256", "use": "sig",
|
||||
"n": raw(rk.N.Bytes()),
|
||||
"e": raw(big.NewInt(int64(rk.E)).Bytes()),
|
||||
},
|
||||
{
|
||||
"kty": "EC", "kid": "ec-1", "alg": "ES256", "use": "sig", "crv": "P-256",
|
||||
"x": raw(ek.X.Bytes()), "y": raw(ek.Y.Bytes()),
|
||||
},
|
||||
}})
|
||||
})
|
||||
idp.Server = httptest.NewServer(mux)
|
||||
t.Cleanup(idp.Close)
|
||||
return idp
|
||||
}
|
||||
|
||||
func raw(b []byte) string { return base64.RawURLEncoding.EncodeToString(b) }
|
||||
|
||||
func (i *testIdP) sign(t *testing.T, alg, kid string, claims map[string]any) string {
|
||||
t.Helper()
|
||||
h, _ := json.Marshal(map[string]string{"alg": alg, "kid": kid, "typ": "JWT"})
|
||||
p, _ := json.Marshal(claims)
|
||||
signing := raw(h) + "." + raw(p)
|
||||
digest := sha256.Sum256([]byte(signing))
|
||||
|
||||
var sig []byte
|
||||
switch alg {
|
||||
case "RS256":
|
||||
s, err := rsa.SignPKCS1v15(rand.Reader, i.rsaKey, crypto.SHA256, digest[:])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sig = s
|
||||
case "ES256":
|
||||
r, s, err := ecdsa.Sign(rand.Reader, i.ecKey, digest[:])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// JWS wants fixed-width r||s, not ASN.1.
|
||||
sig = make([]byte, 64)
|
||||
r.FillBytes(sig[:32])
|
||||
s.FillBytes(sig[32:])
|
||||
default:
|
||||
t.Fatalf("unsupported test alg %q", alg)
|
||||
}
|
||||
return signing + "." + raw(sig)
|
||||
}
|
||||
|
||||
func (i *testIdP) claims(extra map[string]any) map[string]any {
|
||||
c := map[string]any{
|
||||
"iss": i.URL, "sub": "user-1", "aud": "echolot",
|
||||
"exp": time.Now().Add(time.Hour).Unix(), "iat": time.Now().Unix(),
|
||||
"email": "someone@example.net", "groups": []string{"users"},
|
||||
}
|
||||
for k, v := range extra {
|
||||
c[k] = v
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func verifier(i *testIdP, adminGroup string) *Verifier {
|
||||
return New(Config{
|
||||
Issuer: i.URL, ClientID: "echolot", AppClientID: "echolot-app", AdminGroup: adminGroup,
|
||||
}, i.Client())
|
||||
}
|
||||
|
||||
func TestAcceptsAGenuineToken(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
for _, tc := range []struct{ alg, kid string }{{"RS256", "rsa-1"}, {"ES256", "ec-1"}} {
|
||||
got, err := v.Verify(context.Background(), idp.sign(t, tc.alg, tc.kid, idp.claims(nil)))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", tc.alg, err)
|
||||
}
|
||||
if got.Subject != "user-1" || got.Email != "someone@example.net" {
|
||||
t.Fatalf("%s: claims not parsed: %+v", tc.alg, got)
|
||||
}
|
||||
if want := idp.URL + "#user-1"; got.AccountID() != want {
|
||||
t.Errorf("AccountID = %q, want %q", got.AccountID(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// "alg": "none" is the oldest JWT forgery there is: strip the signature, declare no algorithm,
|
||||
// and a naive verifier accepts anything. It must not even reach the key lookup.
|
||||
func TestRejectsAlgNone(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
h, _ := json.Marshal(map[string]string{"alg": "none", "kid": "rsa-1", "typ": "JWT"})
|
||||
p, _ := json.Marshal(idp.claims(nil))
|
||||
token := raw(h) + "." + raw(p) + "."
|
||||
if _, err := v.Verify(context.Background(), token); !errors.Is(err, ErrSignature) {
|
||||
t.Fatalf("alg=none was not refused as a signature failure: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The other classic: declare HS256 so the verifier treats the RSA *public* key as an HMAC secret,
|
||||
// which the attacker also has. The allow-list has no symmetric algorithms at all.
|
||||
func TestRejectsSymmetricAlgorithmConfusion(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
h, _ := json.Marshal(map[string]string{"alg": "HS256", "kid": "rsa-1", "typ": "JWT"})
|
||||
p, _ := json.Marshal(idp.claims(nil))
|
||||
token := raw(h) + "." + raw(p) + "." + raw([]byte("whatever"))
|
||||
if _, err := v.Verify(context.Background(), token); !errors.Is(err, ErrSignature) {
|
||||
t.Fatalf("HS256 confusion was not refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRejectsATamperedPayload(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
good := idp.sign(t, "RS256", "rsa-1", idp.claims(nil))
|
||||
// Swap the payload for one claiming to be somebody else, keeping the valid signature.
|
||||
forged, _ := json.Marshal(idp.claims(map[string]any{"sub": "admin"}))
|
||||
parts := []byte(good)
|
||||
dot1, dot2 := 0, 0
|
||||
for i, c := range parts {
|
||||
if c == '.' {
|
||||
if dot1 == 0 {
|
||||
dot1 = i
|
||||
} else {
|
||||
dot2 = i
|
||||
}
|
||||
}
|
||||
}
|
||||
token := string(parts[:dot1+1]) + raw(forged) + string(parts[dot2:])
|
||||
if _, err := v.Verify(context.Background(), token); !errors.Is(err, ErrSignature) {
|
||||
t.Fatalf("a swapped payload was not refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A token from the same IdP but issued to a different client is perfectly valid — just not for
|
||||
// us. Accepting it would let any other client of the same provider authenticate here.
|
||||
func TestRejectsAnotherClientsToken(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
tok := idp.sign(t, "RS256", "rsa-1", idp.claims(map[string]any{"aud": "some-other-app"}))
|
||||
if _, err := v.Verify(context.Background(), tok); !errors.Is(err, ErrClaims) {
|
||||
t.Fatalf("another client's token was accepted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAcceptsAudienceArrayContainingUs(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
tok := idp.sign(t, "RS256", "rsa-1", idp.claims(map[string]any{"aud": []string{"other", "echolot"}}))
|
||||
if _, err := v.Verify(context.Background(), tok); err != nil {
|
||||
t.Fatalf("an audience array including us was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRejectsExpiredAndFutureTokens(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
expired := idp.sign(t, "RS256", "rsa-1", idp.claims(map[string]any{
|
||||
"exp": time.Now().Add(-time.Hour).Unix(),
|
||||
}))
|
||||
if _, err := v.Verify(context.Background(), expired); !errors.Is(err, ErrClaims) {
|
||||
t.Errorf("expired token accepted: %v", err)
|
||||
}
|
||||
future := idp.sign(t, "RS256", "rsa-1", idp.claims(map[string]any{
|
||||
"iat": time.Now().Add(time.Hour).Unix(),
|
||||
}))
|
||||
if _, err := v.Verify(context.Background(), future); !errors.Is(err, ErrClaims) {
|
||||
t.Errorf("token issued in the future accepted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A token signed by a completely different provider, with its own keys and its own kid.
|
||||
func TestRejectsATokenFromAnotherIssuer(t *testing.T) {
|
||||
ours, theirs := newIdP(t), newIdP(t)
|
||||
v := verifier(ours, "")
|
||||
tok := theirs.sign(t, "RS256", "rsa-1", theirs.claims(nil))
|
||||
if _, err := v.Verify(context.Background(), tok); err == nil {
|
||||
t.Fatal("a token from another issuer was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRejectsMalformedTokens(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
for _, bad := range []string{"", "not-a-token", "a.b", "a.b.c.d", "...", "!!!.???.***"} {
|
||||
if _, err := v.Verify(context.Background(), bad); err == nil {
|
||||
t.Errorf("%q was accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// With no admin group configured, nobody is an admin. An operator who has not said who may
|
||||
// administer the server has not thereby said "anyone who can log in".
|
||||
func TestNobodyIsAdminUntilAGroupIsConfigured(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
claims := &Claims{Groups: []string{"users", "echolot-admins"}}
|
||||
|
||||
if verifier(idp, "").IsAdmin(claims) {
|
||||
t.Error("someone was an admin with no admin group configured")
|
||||
}
|
||||
if !verifier(idp, "echolot-admins").IsAdmin(claims) {
|
||||
t.Error("a member of the configured group was not an admin")
|
||||
}
|
||||
if verifier(idp, "other-group").IsAdmin(claims) {
|
||||
t.Error("a non-member was an admin")
|
||||
}
|
||||
if verifier(idp, "echolot-admins").IsAdmin(nil) {
|
||||
t.Error("an absent identity was an admin")
|
||||
}
|
||||
}
|
||||
|
||||
// A discovery document whose issuer disagrees with the configured one means we were redirected
|
||||
// somewhere — and would otherwise have fetched that somewhere's signing keys while believing
|
||||
// they belonged to the configured provider.
|
||||
func TestRefusesDiscoveryThatRenamesTheIssuer(t *testing.T) {
|
||||
mux := http.NewServeMux()
|
||||
srv := httptest.NewServer(mux)
|
||||
defer srv.Close()
|
||||
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, r *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(Discovery{Issuer: "https://somewhere.else", JWKSURI: srv.URL + "/jwks"})
|
||||
})
|
||||
v := New(Config{Issuer: srv.URL, ClientID: "echolot"}, srv.Client())
|
||||
if _, err := v.Discover(context.Background()); err == nil {
|
||||
t.Fatal("discovery accepted a document for a different issuer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisabledWithoutConfiguration(t *testing.T) {
|
||||
v := New(Config{}, nil)
|
||||
if v.Config().Enabled() {
|
||||
t.Fatal("an unconfigured verifier reports itself enabled")
|
||||
}
|
||||
if _, err := v.Verify(context.Background(), "x.y.z"); !errors.Is(err, ErrDisabled) {
|
||||
t.Fatalf("want ErrDisabled, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Two clients, because the phone and the admin UI have different properties: an APK cannot keep a
|
||||
// secret (public + PKCE) while the server can (confidential). Both must be accepted — but only
|
||||
// those two. "Any client of this issuer" would let every other application registered with the
|
||||
// same IdP authenticate here, which is the whole reason the audience check exists.
|
||||
func TestBothRegisteredClientsAreAccepted(t *testing.T) {
|
||||
idp := newIdP(t)
|
||||
v := verifier(idp, "")
|
||||
|
||||
for _, aud := range []any{"echolot", "echolot-app", []string{"echolot-app", "other"}} {
|
||||
tok := idp.sign(t, "RS256", "rsa-1", idp.claims(map[string]any{"aud": aud}))
|
||||
if _, err := v.Verify(context.Background(), tok); err != nil {
|
||||
t.Errorf("aud %v was refused: %v", aud, err)
|
||||
}
|
||||
}
|
||||
// A third application at the same issuer is still not us.
|
||||
tok := idp.sign(t, "RS256", "rsa-1", idp.claims(map[string]any{"aud": "someone-elses-app"}))
|
||||
if _, err := v.Verify(context.Background(), tok); !errors.Is(err, ErrClaims) {
|
||||
t.Fatalf("a third client's token was accepted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Either client id alone is enough to make sign-in usable: an operator may register only the app
|
||||
// (no admin UI login) or only the server.
|
||||
func TestEitherClientIDAloneEnablesSignIn(t *testing.T) {
|
||||
if !(Config{Issuer: "https://i", ClientID: "a"}).Enabled() {
|
||||
t.Error("a server-only configuration was reported disabled")
|
||||
}
|
||||
if !(Config{Issuer: "https://i", AppClientID: "b"}).Enabled() {
|
||||
t.Error("an app-only configuration was reported disabled")
|
||||
}
|
||||
if (Config{Issuer: "https://i"}).Enabled() {
|
||||
t.Error("an issuer with no client at all was reported enabled")
|
||||
}
|
||||
}
|
||||
@@ -38,10 +38,9 @@ const (
|
||||
// ModeAnonymous accepts uploads from any enrolled device. The default: enrollment already
|
||||
// required an admin-minted token, so "anyone enrolled" is not "anyone".
|
||||
ModeAnonymous Mode = "anonymous"
|
||||
// ModeAccount accepts uploads only from a device tied to a signed-in account. The account
|
||||
// system (OIDC) is not built yet, so today this refuses everything with a distinct reason —
|
||||
// it exists so operators can pick the strict setting now and have it mean the right thing
|
||||
// when accounts land, rather than silently loosening on upgrade.
|
||||
// ModeAccount accepts uploads only from a device where somebody has signed in (see
|
||||
// /v1/account/link). Enrollment alone is not enough: the operator's token admits a device,
|
||||
// an account attributes it to a person.
|
||||
ModeAccount Mode = "account"
|
||||
)
|
||||
|
||||
@@ -120,22 +119,27 @@ func Open(stateDir string, p Policy) (*Store, error) {
|
||||
|
||||
func (s *Store) Policy() Policy { return s.policy }
|
||||
|
||||
// Accepts reports whether an upload would be allowed at all, so callers can answer the
|
||||
// capability question without a body.
|
||||
func (s *Store) Accepts() error {
|
||||
// Accepts reports whether an upload from this caller would be allowed at all, so callers can
|
||||
// answer the capability question without a body.
|
||||
//
|
||||
// linked says whether a person has signed in on the uploading device. It is the only thing that
|
||||
// distinguishes ModeAccount from ModeOff — and the reason the check takes an argument at all.
|
||||
func (s *Store) Accepts(linked bool) error {
|
||||
switch s.policy.Mode {
|
||||
case ModeOff:
|
||||
return ErrDisabled
|
||||
case ModeAccount:
|
||||
if !linked {
|
||||
return ErrNeedAccount
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Put validates and stores one uploaded document. body is the raw JSON as received: it is stored
|
||||
// byte-for-byte so what the device signed off on is what sits on disk.
|
||||
func (s *Store) Put(deviceID string, body []byte) (Meta, error) {
|
||||
if err := s.Accepts(); err != nil {
|
||||
func (s *Store) Put(deviceID string, body []byte, linked bool) (Meta, error) {
|
||||
if err := s.Accepts(linked); err != nil {
|
||||
return Meta{}, err
|
||||
}
|
||||
if s.policy.MaxBytes > 0 && int64(len(body)) > s.policy.MaxBytes {
|
||||
|
||||
@@ -34,19 +34,33 @@ func TestModeOffRefusesEverything(t *testing.T) {
|
||||
p := DefaultPolicy()
|
||||
p.Mode = ModeOff
|
||||
s, _ := open(t, p)
|
||||
if _, err := s.Put("dev1", doc("run-1", AnonFull)); !errors.Is(err, ErrDisabled) {
|
||||
if _, err := s.Put("dev1", doc("run-1", AnonFull), false); !errors.Is(err, ErrDisabled) {
|
||||
t.Fatalf("want ErrDisabled, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ModeAccount must refuse today rather than fall back to anonymous: an operator who selects the
|
||||
// strict setting before accounts exist must not be silently running the permissive one.
|
||||
func TestModeAccountRefusesUntilAccountsExist(t *testing.T) {
|
||||
// ModeAccount turns on whether the *caller* has signed in, and nothing else. A device that has
|
||||
// not is refused with a reason it can act on; one that has is treated exactly like anonymous mode.
|
||||
func TestModeAccountTurnsOnWhetherTheCallerSignedIn(t *testing.T) {
|
||||
p := DefaultPolicy()
|
||||
p.Mode = ModeAccount
|
||||
s, _ := open(t, p)
|
||||
if _, err := s.Put("dev1", doc("run-1", AnonFull)); !errors.Is(err, ErrNeedAccount) {
|
||||
t.Fatalf("want ErrNeedAccount, got %v", err)
|
||||
|
||||
if _, err := s.Put("dev1", doc("run-1", AnonFull), false); !errors.Is(err, ErrNeedAccount) {
|
||||
t.Fatalf("an un-signed-in device was not refused: %v", err)
|
||||
}
|
||||
if _, err := s.Put("dev1", doc("run-2", AnonFull), true); err != nil {
|
||||
t.Fatalf("a signed-in device was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Signing in must not open a door that the operator closed outright: mode=off means off.
|
||||
func TestSigningInDoesNotOverrideModeOff(t *testing.T) {
|
||||
p := DefaultPolicy()
|
||||
p.Mode = ModeOff
|
||||
s, _ := open(t, p)
|
||||
if _, err := s.Put("dev1", doc("run-1", AnonFull), true); !errors.Is(err, ErrDisabled) {
|
||||
t.Fatalf("a signed-in device uploaded to a server with uploads off: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,15 +69,15 @@ func TestMinAnonymizationEnforced(t *testing.T) {
|
||||
p.MinAnonymization = AnonBalanced
|
||||
s, _ := open(t, p)
|
||||
|
||||
if _, err := s.Put("dev1", doc("run-full", AnonFull)); !errors.Is(err, ErrNotAnonEnough) {
|
||||
if _, err := s.Put("dev1", doc("run-full", AnonFull), false); !errors.Is(err, ErrNotAnonEnough) {
|
||||
t.Fatalf("full should be refused when balanced is required, got %v", err)
|
||||
}
|
||||
// An undeclared level means nothing was stripped, so it must be treated as "full".
|
||||
if _, err := s.Put("dev1", []byte(`{"run":{"id":"run-bare"},"summary":{}}`)); !errors.Is(err, ErrNotAnonEnough) {
|
||||
if _, err := s.Put("dev1", []byte(`{"run":{"id":"run-bare"},"summary":{}}`), false); !errors.Is(err, ErrNotAnonEnough) {
|
||||
t.Fatalf("undeclared level should be treated as full, got %v", err)
|
||||
}
|
||||
for _, lvl := range []string{AnonBalanced, AnonStrict} {
|
||||
if _, err := s.Put("dev1", doc("run-"+lvl, lvl)); err != nil {
|
||||
if _, err := s.Put("dev1", doc("run-"+lvl, lvl), false); err != nil {
|
||||
t.Fatalf("%s should be accepted: %v", lvl, err)
|
||||
}
|
||||
}
|
||||
@@ -74,7 +88,7 @@ func TestSizeLimit(t *testing.T) {
|
||||
p.MaxBytes = 200
|
||||
s, _ := open(t, p)
|
||||
big := append(doc("run-1", AnonFull), make([]byte, 400)...)
|
||||
if _, err := s.Put("dev1", big); !errors.Is(err, ErrTooLarge) {
|
||||
if _, err := s.Put("dev1", big, false); !errors.Is(err, ErrTooLarge) {
|
||||
t.Fatalf("want ErrTooLarge, got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -84,7 +98,7 @@ func TestRetentionByCountKeepsNewest(t *testing.T) {
|
||||
p.MaxRunsPerDevice = 3
|
||||
s, _ := open(t, p)
|
||||
for i := 0; i < 6; i++ {
|
||||
if _, err := s.Put("dev1", doc(fmt.Sprintf("run-%d", i), AnonFull)); err != nil {
|
||||
if _, err := s.Put("dev1", doc(fmt.Sprintf("run-%d", i), AnonFull), false); err != nil {
|
||||
t.Fatalf("put %d: %v", i, err)
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond) // distinct UploadedAt so "newest" is well defined
|
||||
@@ -109,7 +123,7 @@ func TestRetentionByAge(t *testing.T) {
|
||||
p.RetentionDays = 7
|
||||
p.MaxRunsPerDevice = 0
|
||||
s, dir := open(t, p)
|
||||
if _, err := s.Put("dev1", doc("run-old", AnonFull)); err != nil {
|
||||
if _, err := s.Put("dev1", doc("run-old", AnonFull), false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Backdate the index entry past the retention window.
|
||||
@@ -123,7 +137,7 @@ func TestRetentionByAge(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if _, err := s.Put("dev1", doc("run-new", AnonFull)); err != nil {
|
||||
if _, err := s.Put("dev1", doc("run-new", AnonFull), false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := s.List("dev1")
|
||||
@@ -136,7 +150,7 @@ func TestRetentionByAge(t *testing.T) {
|
||||
// store directory or overwrite another device's data.
|
||||
func TestIDsCannotEscapeTheStoreDirectory(t *testing.T) {
|
||||
s, dir := open(t, DefaultPolicy())
|
||||
if _, err := s.Put("../../etc", doc("../../../passwd", AnonFull)); err != nil {
|
||||
if _, err := s.Put("../../etc", doc("../../../passwd", AnonFull), false); err != nil {
|
||||
t.Fatalf("put: %v", err)
|
||||
}
|
||||
var found []string
|
||||
@@ -159,10 +173,10 @@ func TestIDsCannotEscapeTheStoreDirectory(t *testing.T) {
|
||||
|
||||
func TestListIsPerDevice(t *testing.T) {
|
||||
s, _ := open(t, DefaultPolicy())
|
||||
if _, err := s.Put("devA", doc("run-a", AnonFull)); err != nil {
|
||||
if _, err := s.Put("devA", doc("run-a", AnonFull), false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Put("devB", doc("run-b", AnonFull)); err != nil {
|
||||
if _, err := s.Put("devB", doc("run-b", AnonFull), false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := s.List("devA"); len(got) != 1 || got[0].ID != "run-a" {
|
||||
@@ -175,7 +189,7 @@ func TestListIsPerDevice(t *testing.T) {
|
||||
|
||||
func TestMetaSummarisesTheDocument(t *testing.T) {
|
||||
s, _ := open(t, DefaultPolicy())
|
||||
m, err := s.Put("dev1", doc("run-1", AnonBalanced))
|
||||
m, err := s.Put("dev1", doc("run-1", AnonBalanced), false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -190,7 +204,7 @@ func TestMetaSummarisesTheDocument(t *testing.T) {
|
||||
func TestMalformedRejected(t *testing.T) {
|
||||
s, _ := open(t, DefaultPolicy())
|
||||
for _, body := range [][]byte{[]byte("not json"), []byte(`{"run":{}}`), []byte(`{}`)} {
|
||||
if _, err := s.Put("dev1", body); !errors.Is(err, ErrMalformed) {
|
||||
if _, err := s.Put("dev1", body, false); !errors.Is(err, ErrMalformed) {
|
||||
t.Fatalf("body %q: want ErrMalformed, got %v", body, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ package selfupdate
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"echo-lot.app/server/internal/system"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
@@ -132,6 +133,15 @@ func Run(api, currentVersion string) error {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("atomic replace failed (filesystem boundaries?): %w", err)
|
||||
}
|
||||
// Serving became an explicit verb, and a unit written before that change starts this binary
|
||||
// with no arguments - which now prints usage and exits non-zero. The unit is not part of what
|
||||
// an update replaces, so it is repaired here rather than left to fail at the next restart,
|
||||
// which might be a reboot months from now.
|
||||
if repaired, err := system.RepairExecStart(); err != nil {
|
||||
fmt.Println("WARNING: could not update the systemd unit for --serve:", err)
|
||||
} else if repaired {
|
||||
fmt.Println("updated the systemd unit to pass --serve (serving is now an explicit verb)")
|
||||
}
|
||||
fmt.Printf("updated %s -> %s (%s); restart to run it\n", currentVersion, rel.TagName, self)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -41,6 +41,7 @@ type Session struct {
|
||||
udpObs []UDPObservation // ring, newest last, cap obsCap
|
||||
connectBack []ConnectBackResult
|
||||
throughput []ThroughputReport
|
||||
upstream UpstreamCounter
|
||||
}
|
||||
|
||||
const obsCap = 4096
|
||||
@@ -67,6 +68,36 @@ type ThroughputReport struct {
|
||||
LimitedBy string `json:"limited_by"`
|
||||
}
|
||||
|
||||
// UpstreamCounter is the server's tally of a client-driven throughput run.
|
||||
//
|
||||
// Deliberately a counter and not a list. A five-second upstream run at 20 Mbps is around ten
|
||||
// thousand packets; one observation struct each would turn a measurement into an allocation
|
||||
// storm on a shared server, and nothing downstream needs the per-packet detail - the client
|
||||
// already has its own send record. The gap between the two counts IS the loss.
|
||||
type UpstreamCounter struct {
|
||||
Packets int `json:"packets"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
FirstRxNs int64 `json:"first_rx_ns"`
|
||||
LastRxNs int64 `json:"last_rx_ns"`
|
||||
}
|
||||
|
||||
// SpanMs is the time between the first and last packet, which is the interval the rate should be
|
||||
// computed over - not the client's requested duration, which includes ramp-up and the tail.
|
||||
func (u UpstreamCounter) SpanMs() int64 {
|
||||
if u.Packets < 2 || u.LastRxNs <= u.FirstRxNs {
|
||||
return 0
|
||||
}
|
||||
return (u.LastRxNs - u.FirstRxNs) / 1_000_000
|
||||
}
|
||||
|
||||
// Kbps is bits per millisecond, which is kilobits per second - no scaling constant to get wrong.
|
||||
func (u UpstreamCounter) Kbps() int {
|
||||
if ms := u.SpanMs(); ms > 0 {
|
||||
return int(u.Bytes * 8 / ms)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ConnectBackResult records one connect-back action outcome.
|
||||
type ConnectBackResult struct {
|
||||
ActionID string `json:"action_id"`
|
||||
@@ -100,6 +131,36 @@ func (s *Session) Observations() (packetsSeen uint64, udp []UDPObservation, cb [
|
||||
append([]ConnectBackResult(nil), s.connectBack...)
|
||||
}
|
||||
|
||||
// CountUpstream tallies one client-sent throughput packet.
|
||||
//
|
||||
// Called on the hot path for every packet of an upstream run, so it does exactly two additions
|
||||
// and two comparisons under the lock and allocates nothing.
|
||||
func (s *Session) CountUpstream(sizeBytes int, tRxNs int64) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.upstream.Packets == 0 {
|
||||
s.upstream.FirstRxNs = tRxNs
|
||||
}
|
||||
s.upstream.Packets++
|
||||
s.upstream.Bytes += int64(sizeBytes)
|
||||
s.upstream.LastRxNs = tRxNs
|
||||
}
|
||||
|
||||
// Upstream returns the tally so far.
|
||||
func (s *Session) Upstream() UpstreamCounter {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.upstream
|
||||
}
|
||||
|
||||
// ResetUpstream clears the tally, so a second run in one session measures itself rather than
|
||||
// inheriting the first one's packets.
|
||||
func (s *Session) ResetUpstream() {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.upstream = UpstreamCounter{}
|
||||
}
|
||||
|
||||
// RecordThroughput stores the server's account of one sustained send.
|
||||
//
|
||||
// Kept as a per-action summary rather than per-packet records: a ten-second run at 50 Mbps is
|
||||
|
||||
@@ -16,6 +16,8 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
|
||||
"echo-lot.app/server/internal/adminauth"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -37,8 +39,19 @@ type Device struct {
|
||||
Credential string `json:"credential"`
|
||||
Enrolled time.Time `json:"enrolled"`
|
||||
Name string `json:"name,omitempty"`
|
||||
|
||||
// The account this device belongs to, as issuer#subject — empty when nobody has signed in
|
||||
// on it. Enrollment and sign-in are deliberately separate: a device is admitted by an
|
||||
// operator's token, and only later (if ever) associated with a person. Servers that accept
|
||||
// anonymous uploads never need the second step.
|
||||
AccountID string `json:"account_id,omitempty"`
|
||||
AccountName string `json:"account_name,omitempty"`
|
||||
LinkedAt time.Time `json:"linked_at,omitempty"`
|
||||
}
|
||||
|
||||
// LinkedToAccount reports whether a person has signed in on this device.
|
||||
func (d Device) LinkedToAccount() bool { return d.AccountID != "" }
|
||||
|
||||
type Store struct {
|
||||
mu sync.Mutex
|
||||
path string
|
||||
@@ -48,6 +61,12 @@ type Store struct {
|
||||
type fileData struct {
|
||||
Tokens []EnrollToken `json:"tokens"`
|
||||
Devices []Device `json:"devices"`
|
||||
// The break-glass admin. Absent until an operator sets one.
|
||||
LocalAdmin *adminauth.Credential `json:"local_admin,omitempty"`
|
||||
// Signing secret for admin session cookies. Persisted so sessions survive a restart;
|
||||
// deleting it from the state file invalidates every session at once, which is how an
|
||||
// operator revokes them.
|
||||
SessionSecret string `json:"session_secret,omitempty"`
|
||||
}
|
||||
|
||||
func Open(stateDir string) (*Store, error) {
|
||||
@@ -126,6 +145,90 @@ func (s *Store) Redeem(token, name string) (*Device, error) {
|
||||
}
|
||||
|
||||
// DeviceByCredential authenticates a bearer credential.
|
||||
// LinkAccount ties a device to a signed-in identity, or clears it when accountID is empty.
|
||||
func (s *Store) LinkAccount(deviceID, accountID, displayName string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
for i := range s.data.Devices {
|
||||
if s.data.Devices[i].ID != deviceID {
|
||||
continue
|
||||
}
|
||||
s.data.Devices[i].AccountID = accountID
|
||||
s.data.Devices[i].AccountName = displayName
|
||||
if accountID == "" {
|
||||
s.data.Devices[i].LinkedAt = time.Time{}
|
||||
} else {
|
||||
s.data.Devices[i].LinkedAt = time.Now().UTC()
|
||||
}
|
||||
return s.save()
|
||||
}
|
||||
return errors.New("no such device")
|
||||
}
|
||||
|
||||
// Devices returns a copy of the device list, for the admin UI.
|
||||
func (s *Store) Devices() []Device {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]Device(nil), s.data.Devices...)
|
||||
}
|
||||
|
||||
// DeleteDevice revokes a device: its credential stops working immediately.
|
||||
func (s *Store) DeleteDevice(id string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
for i := range s.data.Devices {
|
||||
if s.data.Devices[i].ID == id {
|
||||
s.data.Devices = append(s.data.Devices[:i], s.data.Devices[i+1:]...)
|
||||
return s.save()
|
||||
}
|
||||
}
|
||||
return errors.New("no such device")
|
||||
}
|
||||
|
||||
// SetLocalAdmin stores (or replaces) the break-glass admin password.
|
||||
func (s *Store) SetLocalAdmin(c adminauth.Credential) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.data.LocalAdmin = &c
|
||||
return s.save()
|
||||
}
|
||||
|
||||
// LocalAdmin returns the configured break-glass admin, or nil.
|
||||
func (s *Store) LocalAdmin() *adminauth.Credential {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.data.LocalAdmin == nil {
|
||||
return nil
|
||||
}
|
||||
c := *s.data.LocalAdmin
|
||||
return &c
|
||||
}
|
||||
|
||||
// ClearLocalAdmin removes the break-glass admin.
|
||||
func (s *Store) ClearLocalAdmin() error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.data.LocalAdmin = nil
|
||||
return s.save()
|
||||
}
|
||||
|
||||
// SessionSecret returns the admin session signing secret, creating one on first use.
|
||||
func (s *Store) SessionSecret() ([]byte, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.data.SessionSecret != "" {
|
||||
if b, err := hex.DecodeString(s.data.SessionSecret); err == nil && len(b) >= 32 {
|
||||
return b, nil
|
||||
}
|
||||
}
|
||||
b, err := adminauth.NewSecret()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.data.SessionSecret = hex.EncodeToString(b)
|
||||
return b, s.save()
|
||||
}
|
||||
|
||||
func (s *Store) DeviceByCredential(cred string) *Device {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
//go:build linux
|
||||
|
||||
package system
|
||||
|
||||
import (
|
||||
"os"
|
||||
"syscall"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
// DisableEcho turns off terminal echo while a password is typed, returning a function that puts
|
||||
// the terminal back. Both are best-effort: when stdin is a pipe (the automation case) there is
|
||||
// no terminal to change and nothing to restore.
|
||||
func DisableEcho(f *os.File) (func(), error) {
|
||||
fd := f.Fd()
|
||||
var t syscall.Termios
|
||||
if _, _, errno := syscall.Syscall6(syscall.SYS_IOCTL, fd,
|
||||
syscall.TCGETS, uintptr(unsafe.Pointer(&t)), 0, 0, 0); errno != 0 {
|
||||
return nil, errno // not a terminal; nothing to do
|
||||
}
|
||||
original := t
|
||||
t.Lflag &^= syscall.ECHO
|
||||
if _, _, errno := syscall.Syscall6(syscall.SYS_IOCTL, fd,
|
||||
syscall.TCSETS, uintptr(unsafe.Pointer(&t)), 0, 0, 0); errno != 0 {
|
||||
return nil, errno
|
||||
}
|
||||
return func() {
|
||||
_, _, _ = syscall.Syscall6(syscall.SYS_IOCTL, fd,
|
||||
syscall.TCSETS, uintptr(unsafe.Pointer(&original)), 0, 0, 0)
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
//go:build !linux
|
||||
|
||||
package system
|
||||
|
||||
import "os"
|
||||
|
||||
// DisableEcho is a no-op off Linux: the password is still read, just echoed. Better than
|
||||
// refusing to run — an operator on a Mac still needs to set the break-glass password.
|
||||
func DisableEcho(*os.File) (func(), error) { return nil, nil }
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -29,7 +30,7 @@ Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=%s
|
||||
ExecStart=%s --serve
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
StateDirectory=echolot-server
|
||||
@@ -144,3 +145,41 @@ func UninstallSystemd() error {
|
||||
fmt.Println("removed echolot-server units (state dir and env file left in place)")
|
||||
return nil
|
||||
}
|
||||
|
||||
// RepairExecStart brings an already-installed unit up to date with the current invocation.
|
||||
//
|
||||
// Serving became an explicit verb (--serve), which means every unit written before that change
|
||||
// would start the binary with no arguments — and the binary now answers that with usage and a
|
||||
// non-zero exit. A self-update replaces the binary but never the unit, so without this a routine
|
||||
// update would leave a service that cannot start, discovered whenever the host next reboots.
|
||||
//
|
||||
// Only a unit this program wrote is touched, identified by its description line. Editing an
|
||||
// operator's hand-written unit would be overreach; leaving ours broken would be negligence.
|
||||
func RepairExecStart() (repaired bool, err error) {
|
||||
b, err := os.ReadFile(unitPath)
|
||||
if err != nil {
|
||||
return false, nil // no unit installed: nothing to repair, and not an error
|
||||
}
|
||||
text := string(b)
|
||||
if !strings.Contains(text, "Echolot probe server") {
|
||||
return false, nil // somebody else's unit
|
||||
}
|
||||
lines := strings.Split(text, "\n")
|
||||
changed := false
|
||||
for i, ln := range lines {
|
||||
t := strings.TrimSpace(ln)
|
||||
// Only the serving unit's ExecStart; the timer's own line already carries its verb.
|
||||
if strings.HasPrefix(t, "ExecStart=") && !strings.Contains(t, "--") {
|
||||
lines[i] = ln + " --serve"
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return false, nil
|
||||
}
|
||||
if err := os.WriteFile(unitPath, []byte(strings.Join(lines, "\n")), 0o644); err != nil {
|
||||
return false, fmt.Errorf("updating %s: %w", unitPath, err)
|
||||
}
|
||||
_ = exec.Command("systemctl", "daemon-reload").Run()
|
||||
return true, nil
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user