app: probe the server this device is enrolled with, not ours

The canary-DNS zone and the STUN host were compiled in as c.echo-lot.app and fmr-1.echo-lot.app, so every copy of the app measured against this particular deployment whatever server its owner had enrolled with. On someone else's install those two tests describe our infrastructure and report the result as a fact about their network. The zone comes from the server's own profile, which has advertised canary_zone all along — the app simply never read it. It is cached in settings because the canary probe runs at device tier, before anything has contacted the control plane, and a probe that had to make a call first would fail on exactly the networks worth measuring. The STUN host is derived from the configured server URL rather than stored, since a second copy of the server's name goes stale the moment someone re-enrolls elsewhere. With no server configured both now report SKIPPED. StunProbe previously would have reported FAILED on a blank host, which reads as a finding about the network when the truth is that no packet was ever sent — the same conflation between "measured nothing" and "measured a fault" that the ICMPv6 finding had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 23:20:52 +02:00
parent e0428b4c84
commit 4aaaa5f5d4
7 changed files with 99 additions and 5 deletions
@@ -35,9 +35,37 @@ data class Run(
    val device: DeviceInfo,
    val tiers: Tiers,
    @SerialName("profiles_used") val profilesUsed: List<String> = emptyList(),
+    val constraints: Constraints = Constraints(),
    val notes: String? = null,
 )

+/**
+ * What limited this run — the counterpart to [Tiers], which records what was available.
+ *
+ * A constrained run is not a failed run, and it is not a normal one either. Without this, a run
+ * taken through a VPN looks exactly like a clean run of a healthy network: the same shape, the
+ * same green verdict, and no way for a reader — or a server aggregating thousands of these — to
+ * know that almost nothing was actually measured.
+ */
+@Serializable
+data class Constraints(
+    /** A VPN held the default route while this ran. */
+    @SerialName("vpn_active") val vpnActive: Boolean = false,
+    /**
+     * Per-network probing was refused by the OS.
+     *
+     * Android blocks `Network.bindSocket()` on the underlying networks whenever a VPN is up, to
+     * stop apps leaking around the tunnel. Every per-network test then measures nothing, so any
+     * conclusion drawn about the wifi or cellular link underneath is unfounded.
+     */
+    @SerialName("per_network_blocked") val perNetworkBlocked: Boolean = false,
+    /** Networks that could not be measured, by id. */
+    @SerialName("unmeasured_networks") val unmeasuredNetworks: List<String> = emptyList(),
+) {
+    /** True when this run's results mean something different from an unconstrained one. */
+    val constrained: Boolean get() = vpnActive || perNetworkBlocked
+}
+
@Serializable
 enum class Trigger {
    @SerialName("manual") MANUAL,
@@ -214,6 +214,20 @@ object FindingRegistry {
        rulesOut = "A working IPv6 setup: SLAAC did not produce a usable address on this link.",
    )

+    /**
+     * A VPN prevented the underlying networks from being measured.
+     *
+     * Reported rather than worked around: Android refuses `Network.bindSocket()` on the networks
+     * beneath a VPN precisely so apps cannot leak around the tunnel, and that is correct
+     * behaviour. What is not acceptable is a run that quietly measures nothing and calls the
+     * result healthy, so this says plainly which networks went unmeasured and why.
+     */
+    val MEASUREMENT_VPN_CONSTRAINED = FindingSpec(
+        "measurement.vpn_constrained", Category.CONNECTIVITY, Severity.INFO,
+        "A VPN was active, so the networks underneath it could not be measured.",
+        rulesOut = "Nothing — this run says little about the underlying network either way.",
+    )
+
    val V6_NO_DEFAULT_ROUTE = FindingSpec(
        "v6.no_default_route", Category.IPV6, Severity.MEDIUM,
        "The device has a global IPv6 address but no IPv6 default route.",
@@ -247,7 +261,7 @@ object FindingRegistry {
        NAT_UDP_REBINDING, NAT_SYMMETRIC,
        THROUGHPUT_NO_DELIVERY, THROUGHPUT_BELOW_OFFERED,
        DNS_ANSWER_REWRITTEN, DNS_AUTHORITATIVE_UNREACHABLE,
-        V6_NO_DEFAULT_ROUTE, V6_ROUTE_WITHOUT_ADDRESS, V6_NO_ICMP_REPLY, V6_NOT_OFFERED,
+        MEASUREMENT_VPN_CONSTRAINED, V6_NO_DEFAULT_ROUTE, V6_ROUTE_WITHOUT_ADDRESS, V6_NO_ICMP_REPLY, V6_NOT_OFFERED,
    )

    private val byCode: Map<String, FindingSpec> = all.associateBy { it.code }
@@ -35,6 +35,9 @@ data class CategorySummary(
 *   (critical|high → red, medium|low → yellow, info/none → green).
 * - A category is `inconclusive` when > 50% of its tests are failed/unsupported.
 * - Overall = the worst category light; `inconclusive` only when ALL categories are.
+ * - A run whose per-network probing was blocked is `inconclusive` outright, whatever the
+ *   categories say. The lights describe what the tests found; when the OS refused to let the
+ *   tests run, a green light would describe nothing at all.
 *
 * The mapping test-type → category comes from [TestType.category]. Only categories that have
 * findings or tests appear in the summary.
@@ -44,7 +47,10 @@ object Verdicts {
    private fun isInconclusiveTest(s: TestStatus) =
        s == TestStatus.FAILED || s == TestStatus.UNSUPPORTED

-    fun derive(tests: List<Test>, findings: List<Finding>): Summary {
+    fun derive(tests: List<Test>, findings: List<Finding>): Summary =
+        derive(tests, findings, Constraints())
+
+    fun derive(tests: List<Test>, findings: List<Finding>, constraints: Constraints): Summary {
        val testsByCat = tests.groupBy { TestType.category(it.type) }
        val findingsByCat = findings.groupBy { it.category }
        val categories = (testsByCat.keys + findingsByCat.keys)
@@ -72,7 +78,14 @@ object Verdicts {
            )
        }

-        val overall = deriveOverall(perCat.values)
+        // A run that could not measure the networks it was asked about has not found them
+        // healthy; it has found out nothing. Reporting that as green is the single most
+        // misleading thing this function could do, so the constraint outranks the lights.
+        val overall = if (constraints.perNetworkBlocked) {
+            Verdict.INCONCLUSIVE
+        } else {
+            deriveOverall(perCat.values)
+        }
        return Summary(overall = overall, categories = perCat)
    }