findings: a registry, because the codes had already drifted

A finding code is the stable half of a result - what a dashboard groups by and
what someone greps a year of archived runs for. That only holds if a code means
exactly one thing forever, which fifteen ad-hoc string literals cannot promise.

By the time this was written the failure had happened twice:

  - Two emitters independently produced connectivity.downstream_loss and
    connectivity.loss_downstream for the same claim. Nothing objected. Anyone
    aggregating either would have silently seen half their data.
  - Two codes sat under nat.* while being declared Category.CONNECTIVITY.
    nat.udp_unreachable is not about NAT, and the prefix decides the category,
    which decides which verdict light the finding rolls up into. Renamed while
    that is still cheap.

Codes are now typed FindingSpecs carrying category and default severity;
emitters reference the spec rather than retyping the string, so a typo is a
compile error and two call sites cannot disagree about a finding's category.

docs/findings-registry.md is the contract and a test reads it, failing when the
document and the code disagree on which codes exist or how severe they are.
Documentation that drifts from its implementation is worse than none, because it
still looks authoritative. The check reads table rows only, so the prose can go
on explaining which codes were retired and why.

Closes open item 1 of measurement-schema.md section 9.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mrambossek
2026-08-01 14:25:54 +02:00
co-authored by Claude Fable 5
parent f7701c2d2f
commit e7afc2210f
8 changed files with 475 additions and 22 deletions
+29
View File
@@ -819,3 +819,32 @@ prevent. The pre-existing rate test caught that when I first tried the generous
right to. Second half of the same bug: callers treated *any* refusal as terminal, so `TryAllow` now
says why — a sender paces through a transient "too fast just now" and still stops dead on a spent
budget or an expired grant. Both halves are pinned by regression tests.
### Findings registry (2026-08-01)
Closes open item 1 of measurement-schema.md §9. A finding code is the stable, machine-readable half
of a result — what a dashboard groups by and what someone greps a year of archived runs for — and
that only holds if a code means exactly one thing forever. Ad-hoc string literals at fifteen call
sites cannot promise that, and by the time the registry was written the failure had already
happened.
**Two emitters had independently produced `connectivity.downstream_loss` and
`connectivity.loss_downstream` for the same claim**, and nothing anywhere objected. Anyone
aggregating either one would have silently seen half their data. Merged into
`connectivity.loss_downstream`, paired with `loss_upstream` so the two directions read as a set.
**Two codes were also renamed out of `nat.*`.** `nat.udp_unreachable` is not about NAT — it means
no replies came back — but the prefix determines the category, and the category determines which
verdict light the finding rolls up into (§7.3). A `nat.*` code landing under *connectivity* is not
a naming quibble; it changes which light turns red. Cheap to fix now, a breaking change later.
Codes are now declared as typed `FindingSpec`s carrying their category and default severity, and
emitters reference the spec instead of retyping the string — so a typo is a compile error and two
call sites cannot disagree about a finding's category.
`docs/findings-registry.md` is the contract, and a test reads it: it fails when the document and
the registry have codes the other lacks, or when a severity differs. Documentation that drifts from
its implementation is worse than none, because it still looks authoritative. The check scopes
itself to table rows, so the prose can keep explaining which codes were retired and why.
Six tests: uniqueness, declared-vs-listed, prefix↔category agreement, naming convention, a
word-order-anagram check (the shape the duplication actually took), and the document agreement.
+91
View File
@@ -0,0 +1,91 @@
<!--
SPDX-FileCopyrightText: 2026 Echolot contributors
SPDX-License-Identifier: CC-BY-4.0
-->
# Echolot findings registry
Closes open item 1 of `measurement-schema.md` §9.
A **finding code** is the stable, machine-readable half of a result. The prose around it changes
freely; the code is what a dashboard groups by, what a diff between two runs keys on, and what
someone greps a year of archived runs for. That only works if a code means exactly one thing,
forever.
This document is the contract. It is kept in step with
`echolot-app/core-measurement/.../FindingRegistry.kt` by a test that fails when either side has a
code the other does not — a registry that drifts from its documentation is worse than none,
because it looks authoritative.
## Rules
1. **The prefix determines the category**, and the category determines which verdict light the
finding rolls up into (§7.3). A `nat.*` code appearing under *connectivity* is not a naming
quibble; it changes which light turns red. Two codes were renamed from `nat.*` to
`connectivity.*` for exactly this reason.
2. **One code per concept.** Two emitters independently produced `connectivity.downstream_loss`
and `connectivity.loss_downstream` for the same claim before this registry existed. Anyone
aggregating either would have silently seen half their data.
3. **Codes are declared, not typed.** Emitters reference a `FindingSpec`, so a typo is a compile
error and no two call sites can disagree about a finding's category or default severity.
4. **Severity in the registry is the default.** An emitter may escalate for a specific run; it may
not quietly reclassify the finding in general.
5. **Say what is ruled out**, where that is the useful half. "Loss upstream" is worth far more
when it also states that the return path is clean, because that halves where to look next.
6. **Renaming a code is a breaking change** once runs are archived at scale. Before 1.0 it is
cheap; after, it needs an alias and a deprecation window.
## Registry
### connectivity
| code | severity | means | rules out |
|---|---|---|---|
| `connectivity.udp_unreachable` | high | No UDP echo replies came back from the server at all. | — |
| `connectivity.udp_unreachable_upstream` | high | The server received none of the probes, so traffic is dropped on the way out. | The return path: nothing arrived to be replied to. |
| `connectivity.udp_loss` | medium | A large fraction of round-trip probes were lost, direction unknown. | — |
| `connectivity.loss_upstream` | medium | Probes were lost on the way to the server. | The return path: replies came back for everything that arrived. |
| `connectivity.loss_downstream` | medium | Packets were lost on the way back from the server. | The outbound path: the server received what it was answering. |
| `connectivity.downstream_blocked` | high | Server-initiated packets never arrive, although round trips work. | Basic reachability: the path forwards replies, just not unsolicited traffic. |
| `connectivity.downstream_reorder` | low | Downstream packets arrive in a different order than they were sent. | — |
| `connectivity.captive_portal` | high | A captive portal is intercepting connectivity checks. | — |
| `connectivity.no_internet` | high | Android's own connectivity checks fail on this network. | — |
### mtu
| code | severity | means | rules out |
|---|---|---|---|
| `mtu.reduced_downstream` | low | The downstream path MTU is below the usual 1500 bytes. | — |
| `mtu.downstream_blackhole` | medium | Datagrams above the path MTU are dropped downstream, fragmented or not. | — |
| `mtu.fragments_blocked` | medium | IP fragments do not reach this device even when sent in order. | — |
| `mtu.fragment_reorder_sensitive` | low | Fragments are delivered in order but dropped when reordered or delayed. | Fragmentation itself: in-order fragments arrive fine. |
### nat
| code | severity | means | rules out |
|---|---|---|---|
| `nat.udp_rebinding` | medium | A NAT remapped the UDP source port mid-flow. | — |
| `nat.symmetric` | medium | The NAT assigns a different external port per destination. | — |
### perf
| code | severity | means | rules out |
|---|---|---|---|
| `perf.throughput_no_delivery` | high | No throughput traffic arrived, although the server sent it. | — |
| `perf.throughput_below_offered` | low | Less throughput arrived than the server sent for the whole run. | — |
### dns
| code | severity | means | rules out |
|---|---|---|---|
| `dns.answer_rewritten` | high | A resolver returned an answer that differs from the authoritative record. | — |
| `dns.authoritative_unreachable` | medium | The canary zone's authoritative server could not be reached. | — |
## Adding a finding
1. Add a `FindingSpec` to `FindingRegistry`, and to its `all` list.
2. Add the row here, under the section its prefix names.
3. Emit it with `finding(FindingRegistry.YOUR_CODE, …)`.
The registry test checks 1 and 2 agree, that every prefix maps to the category it claims, and that
no two entries share a code.
+2 -1
View File
@@ -279,7 +279,8 @@ Free-text fields (`notes`, `error.detail`, dump excerpts from Shizuku parsers) c
## 9. Open items
1. Findings registry document — start alongside the first implemented tests.
1. ~~Findings registry document~~ — done: `findings-registry.md`, kept in step with
`FindingRegistry.kt` by a test that fails when the two disagree.
2. Whether Shizuku raw-dump excerpts (dumpsys/ip output) are embedded in `evidence` verbatim (auditable, but large and hard to anonymize) or parsed-only with an optional "attach raw dumps" toggle. Proposal: toggle, default on for local archive, default off for export.
3. Peer-mode documents: each device produces its own run; the coordinator embeds the peer's findings summary and cross-references by `run.id`. Full merge format deferred.
4. Size guardrails: soft cap 20 MB uncompressed per run; trains beyond that downsample evidence (keep aggregates + first/last N + all anomalies) and record `"evidence_truncated": true`.