throughput: paced downstream rate, with the qualifier that makes it honest
server-release / image (push) Successful in 15s
server-test / test (push) Successful in 32s
server-release / release (push) Successful in 32s

A throughput number reports the smallest limit on the path, and the sender's own
ceiling is one of the candidates. If the server was asked for 50 Mbps and 50
Mbps arrived, the network was never the constraint and "50 Mbps" says nothing
about it. So the result always carries limited_by and measures_network, and a
finding is raised only when the path is actually implicated.

Loss is computed against the *sender's* count, not the requested rate: the
server reports what it put on the wire, and the gap is the loss. A receiver
alone cannot tell "the network dropped it" from "the sender never sent it", and
guessing turns a healthy server-side limit into a phantom network fault. The
count is stored per action, not per packet — half a million packets of structs
would turn a measurement into memory exhaustion.

Sending is paced rather than flat out. An unpaced burst measures the server's
NIC and the first queue it meets, then collapses into loss that reads as a
network fault. The schedule is absolute rather than sleep-per-packet, which
would accumulate scheduler error and drift the rate down over a ten-second run.

Throughput gets its own grant budget sized from the request, so every other
action stays bounded at 8 MiB. When the byte cap binds before the clock does,
the *duration* is shortened and reported, rather than the run being truncated
halfway: promising thirty seconds and delivering twenty-one is the same
information with a surprise attached, and it keeps "the clock ended the run" as
the normal case — the only case where the rate is a clean property of the path.

That last behaviour came out of a test that failed honestly: 30 s at 100 Mbps
needs 375 MB against a 256 MB cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mrambossek
2026-08-01 14:05:23 +02:00
co-authored by Claude Fable 5
parent 35744c609e
commit 3333788d9e
8 changed files with 581 additions and 2 deletions
+182
View File
@@ -0,0 +1,182 @@
// SPDX-FileCopyrightText: 2026 Echolot contributors
// SPDX-License-Identifier: GPL-3.0-or-later
package dataplane
import (
"encoding/binary"
"fmt"
"time"
"echo-lot.app/server/internal/session"
)
// Sustained-rate sending (spec §5 throughput).
//
// This is the most expensive thing the server will do on a client's say-so, so it is also the
// action where the §3.4 anti-amplification rules matter most. Three bounds apply, and all three
// are enforced here rather than trusted to the caller:
//
// - the destination is the session's *observed* data-plane source, verified by an HMAC-signed
// ECHO that arrived from that address, so this cannot be aimed at a third party;
// - the grant carries a byte budget and an average-rate ceiling, and the send stops the moment
// either is reached;
// - the duration is hard-capped, so a client that vanishes mid-test costs a bounded amount of
// traffic rather than an open-ended one.
//
// The measurement this produces is honest only if the client is told which limit it hit. A run
// that saturates the grant ceiling has measured *us*, not the network, and reporting that as
// throughput would be worse than not measuring at all — see ThroughputResult.LimitedBy.
// ThroughputResult is what the server actually managed to send.
type ThroughputResult struct {
Packets int `json:"packets"`
Bytes int64 `json:"bytes"`
DurationMs int64 `json:"duration_ms"`
Kbps int `json:"kbps"`
// LimitedBy says what stopped it: "duration" (ran the full time, so the rate is the path's
// or ours to give), "budget" (hit the grant's byte ceiling), or "rate" (the pacing ceiling
// held it back). Only "duration" makes the number a property of the network.
LimitedBy string `json:"limited_by"`
}
// ThroughputLimits derives a grant sized for one throughput run.
//
// The default 8 MiB action budget is deliberately far too small for this — ten seconds at
// 50 Mbps is 62 MB — so throughput gets its own budget computed from what it asked for, still
// clamped to a ceiling. Sizing the budget to the request (rather than raising the global default)
// keeps every *other* action bounded at 8 MiB.
func ThroughputLimits(durationMs, kbps int) session.GrantLimits {
durationMs, kbps = ThroughputPlan(durationMs, kbps)
// bytes = kbps * 1000 / 8 * seconds, with a little headroom so the byte budget is not what
// stops a run that was meant to be stopped by the clock.
budget := int64(kbps) * 1000 / 8 * int64(durationMs) / 1000
budget = budget * 11 / 10
if budget > maxThroughputBytes {
budget = maxThroughputBytes
}
return session.GrantLimits{
MaxBytes: budget,
// A little above the pacing target on purpose: the pacer should be what controls the
// rate, and the grant should be the safety net. If they are equal, ordinary scheduling
// jitter trips the grant and the run is cut short for no real reason.
MaxKbps: kbps * 12 / 10,
MaxHold: time.Duration(durationMs)*time.Millisecond + 5*time.Second,
}
}
// ThroughputPlan reduces a request to what this server will actually run, and is the single
// place that decides it.
//
// When the byte cap binds before the clock does, the *duration* is shortened rather than the run
// being cut off partway. Truncating mid-run is not wrong exactly — the rate is still computed
// over the elapsed time and limited_by says "budget" — but it means promising a client thirty
// seconds and giving it twenty-one. Saying "twenty-one seconds" up front is the same information
// without the surprise, and it keeps "the clock ended the run" as the normal case, which is the
// only case where the number is a clean property of the network.
func ThroughputPlan(durationMs, kbps int) (effectiveMs, effectiveKbps int) {
if durationMs <= 0 {
durationMs = 10_000
}
if durationMs > maxThroughputMs {
durationMs = maxThroughputMs
}
if kbps <= 0 || kbps > maxThroughputKbps {
kbps = maxThroughputKbps
}
bytesPerMs := int64(kbps) * 1000 / 8 / 1000
if bytesPerMs > 0 {
if maxMs := maxThroughputBytes / bytesPerMs; int64(durationMs) > maxMs {
durationMs = int(maxMs)
}
}
return durationMs, kbps
}
const (
maxThroughputMs = 30_000
maxThroughputKbps = 200_000
maxThroughputBytes = 256 << 20
)
// DownThroughput sends paced traffic toward the client for up to durationMs.
//
// Pacing is deliberate rather than "send as fast as possible": an unpaced burst measures the
// server's NIC and the first queue it meets, then collapses into loss that looks like a network
// fault. Spacing packets at the target rate makes loss mean what a reader will assume it means.
func (s *Server) DownThroughput(
sess *session.Session, g *session.Grant, durationMs, kbps, sizeBytes int,
) (ThroughputResult, error) {
res := ThroughputResult{}
target := sess.DataSource()
if !target.IsValid() {
return res, fmt.Errorf("no observed data-plane source")
}
conn := s.connFor(target, sess.DataLocal())
if conn == nil {
return res, fmt.Errorf("no data-plane socket matches target family")
}
// Same plan the grant was sized from, so the two cannot disagree.
durationMs, kbps = ThroughputPlan(durationMs, kbps)
if sizeBytes < HeaderSize+16 {
sizeBytes = 1200 // a size that survives every common path unfragmented
}
if sizeBytes > 1472 {
sizeBytes = 1472
}
// Nanoseconds between packets to hit the target rate.
perPacketNs := int64(sizeBytes) * 8 * 1_000_000 / int64(kbps)
if perPacketNs < 1_000 {
perPacketNs = 1_000
}
payload := make([]byte, sizeBytes-HeaderSize)
deadline := time.Now().Add(time.Duration(durationMs) * time.Millisecond)
start := time.Now()
next := start
res.LimitedBy = "duration"
var seq uint32
for time.Now().Before(deadline) {
if !g.Allow(sizeBytes) {
// Distinguishing these two matters: a run stopped by the byte budget has not been
// running long enough for its rate to mean anything.
if g.Sent() >= g.MaxBytes {
res.LimitedBy = "budget"
} else {
res.LimitedBy = "rate"
}
break
}
binary.BigEndian.PutUint32(payload[0:4], seq)
binary.BigEndian.PutUint64(payload[4:12], uint64(time.Since(s.start).Nanoseconds()))
if err := s.sendErr(conn, target, sess, TypeThroughputData, seq, payload); err != nil {
// A send error mid-run is a local condition (buffer full, route gone). Stop and
// report what got out rather than pretending the rest was lost on the path.
res.LimitedBy = "send_error"
break
}
res.Packets++
res.Bytes += int64(sizeBytes)
seq++
// Absolute schedule, not sleep-per-packet: sleeping a fixed interval accumulates the
// scheduler's error and drifts the achieved rate below the target over a 10-second run.
next = next.Add(time.Duration(perPacketNs))
if d := time.Until(next); d > 0 {
time.Sleep(d)
}
}
elapsed := time.Since(start)
res.DurationMs = elapsed.Milliseconds()
// bits per millisecond is kilobits per second, so no scaling constant is needed - and none
// can be got wrong. Guarded because a run that ends inside a millisecond has no rate.
if res.DurationMs > 0 {
res.Kbps = int(res.Bytes * 8 / res.DurationMs)
}
return res, nil
}
@@ -0,0 +1,99 @@
// SPDX-FileCopyrightText: 2026 Echolot contributors
// SPDX-License-Identifier: GPL-3.0-or-later
package dataplane
import (
"testing"
"time"
)
// The grant has to be big enough that the *clock* ends a throughput run, not the byte budget. Get
// this wrong and the test still "works": it stops early, reports a rate computed over a truncated
// window, and nothing anywhere says the number is meaningless. So the sizing is pinned.
func TestThroughputBudgetOutlastsTheRequestedRun(t *testing.T) {
cases := []struct{ durationMs, kbps int }{
{1_000, 1_000},
{10_000, 50_000},
{10_000, 200_000},
{30_000, 100_000},
}
for _, c := range cases {
// Against the *planned* duration, which is what will actually be run: a request the
// server shortens is answered with the shorter number, not truncated halfway.
planMs, planKbps := ThroughputPlan(c.durationMs, c.kbps)
lim := ThroughputLimits(c.durationMs, c.kbps)
needed := int64(planKbps) * 1000 / 8 * int64(planMs) / 1000
if lim.MaxBytes < needed {
t.Errorf("%d ms at %d kbps (planned %d ms) needs %d bytes, budget is %d - the run "+
"would stop early and report a rate over a truncated window",
c.durationMs, c.kbps, planMs, needed, lim.MaxBytes)
}
}
}
// The pacer should control the rate and the grant should be the safety net. If the grant's
// ceiling equals the pacing target, ordinary scheduling jitter trips it and cuts the run short
// for no real reason.
func TestGrantRateCeilingSitsAboveThePacingTarget(t *testing.T) {
lim := ThroughputLimits(10_000, 50_000)
if lim.MaxKbps <= 50_000 {
t.Fatalf("grant ceiling %d kbps is not above the 50000 kbps pacing target", lim.MaxKbps)
}
}
// A client asking for more than the server will do must get the server's number, not its own.
func TestThroughputRequestsAreClamped(t *testing.T) {
lim := ThroughputLimits(10*60*1000, 10_000_000) // ten minutes at 10 Gbps
if lim.MaxBytes > maxThroughputBytes {
t.Errorf("byte budget %d exceeds the hard cap %d", lim.MaxBytes, maxThroughputBytes)
}
if lim.MaxKbps > maxThroughputKbps*12/10 {
t.Errorf("rate ceiling %d exceeds the hard cap", lim.MaxKbps)
}
// The hold has to outlast the planned run, or the grant expires mid-send and the run is
// reported as rate-limited when it was really time-limited.
planMs, _ := ThroughputPlan(10*60*1000, 10_000_000)
if lim.MaxHold < time.Duration(planMs)*time.Millisecond {
t.Errorf("hold %v is shorter than the planned run of %d ms", lim.MaxHold, planMs)
}
}
// When the byte cap binds before the clock does, the server shortens the run and says so, rather
// than accepting thirty seconds and delivering twenty-one. Same information, no surprise - and it
// keeps "the clock ended the run" as the normal case, which is the only case where the resulting
// rate is a clean property of the network.
func TestAnOversizedRequestComesBackShorterRatherThanTruncated(t *testing.T) {
const kbps = 200_000
askedMs := 30_000
planMs, planKbps := ThroughputPlan(askedMs, kbps)
if planKbps != kbps {
t.Errorf("rate was reduced to %d; the duration should absorb the cap, not the rate", planKbps)
}
if planMs >= askedMs {
t.Fatalf("plan kept the full %d ms at %d kbps, which exceeds the %d byte cap",
askedMs, kbps, maxThroughputBytes)
}
// And what it does promise must fit.
if got := int64(planKbps) * 1000 / 8 * int64(planMs) / 1000; got > maxThroughputBytes {
t.Errorf("planned run needs %d bytes, over the %d cap", got, maxThroughputBytes)
}
}
// A short, ordinary request must come back untouched - the clamping only exists for the extremes.
func TestAnOrdinaryRequestIsNotRewritten(t *testing.T) {
planMs, planKbps := ThroughputPlan(10_000, 50_000)
if planMs != 10_000 || planKbps != 50_000 {
t.Errorf("10 s at 50 Mbps was rewritten to %d ms at %d kbps", planMs, planKbps)
}
}
// Every action other than throughput stays on the small default budget. Throughput needs a big
// one; raising the global default to suit it would quietly unbound everything else.
func TestOnlyThroughputGetsTheLargeBudget(t *testing.T) {
big := ThroughputLimits(10_000, 50_000)
if big.MaxBytes <= 8<<20 {
t.Fatalf("throughput budget %d is no larger than the default action budget", big.MaxBytes)
}
}
+2
View File
@@ -37,6 +37,8 @@ const (
TypeBigSend = 0x0C
// TypeFragData is delivered only after IP reassembly, so its arrival IS the measurement.
TypeFragData = 0x0D
// TypeThroughputData is one packet of a sustained-rate downstream run.
TypeThroughputData = 0x0E
)
type Server struct {