Federated relay tier: peer flooding, per-member isolation, registry admission and relay discovery

This commit is contained in:
George Coles
2026-09-15 06:49:43 -04:00
parent d30e612be8
commit 2c97fd523f
11 changed files with 759 additions and 177 deletions
+4 -4
View File
@@ -3,7 +3,7 @@
## Repo shape
- `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
- `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
- Commands: `cargo build`, `cargo test` (80 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
- Commands: `cargo build`, `cargo test` (85 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
- E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.
## Editing the spec
@@ -21,7 +21,7 @@
- Relay addresses transport mailboxes by `key` (unicast `to` = recipient pubkey; queues keyed by pubkey); the identifier is protocol identity only. Registry binding checks `map[key].id == from`.
- Built: envelope/query/response, Tantivy index, aggregates, member directory. Not built: dashboard UI, directory watching, TLS, lineage/delegation. Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
- Economics is out of protocol scope (I4: aggregates advise, contracts govern): no receipt, citation, pricing, or settlement fields or message types exist or may be added.
- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queue drained on poll), and returns 429 + Retry-After under backpressure never silent drops. Mailbox polls require proof of key possession: `GET /v1/challenge` then a signed single-use nonce, so knowing a pubkey is not enough to drain its queue.
- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queues drained on poll), and requires challengeresponse proof of key possession for mailbox polls. Per-member queues are isolated: a lagging member gets 429 + Retry-After with a `missed` count; publishers and other members are never stalled. Relays MAY flood to configured peers (`/v1/federation`, `--peer`+`--url`, hop-bounded, seen-set dedup without suppressing identical direct publishes) and MAY gate senders against a registry (`--registry`+`--ma-key`).
- Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits results with honest `truncated`/`more_available` and no scores (I6). BM25 order is a local implementation detail, not protocol surface.
- Index layout: Tantivy at `<data_dir>/index`, collections manifest at `<data_dir>/collections.toml`; `exposure` (metadata|full) gates whether `content` is returned.
- Egress checks live in the responder path (`src/node.rs` `respond`), not the relay — keep private collections unreachable there.
@@ -30,11 +30,11 @@
- Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale).
- Member authority (Draft 0.5 §6): the MA-signed registry snapshot is authoritative when configured (`[node] registry` = file path or URL, `ma_key` pinned; monotonic version — rollback and forgery close the node; file path is mtime-reloaded, URL is fetched at start + every 60s and cached to `<data_dir>/registry-cache.json`, so outage fails static). Keys carry optional validity windows (`not_before`/`not_after`); rotation = `registry add-key` then `revoke-key`.
- `<data_dir>/members.toml` (name, pubkey, class, `previous` keys, mtime-reloaded) is a dev/local fallback used only when no registry is configured; empty directory without a registry is open bootstrap only when `dev_bootstrap = true` (RFC §6: explicit dev flag). Receivers drop content-bearing responses from enrichment-class senders (metadata-only, §6).
- MA tooling: `frxd registry init|add|add-key|revoke-key|remove|list|show|serve` (signed `registry.json` + `ma-key.hex` in `--dir`); `frxd init --id/--registry/--ma-key`; `frxd key show|rotate`; `member add --previous <old>` for the fallback path.
- MA tooling: `frxd registry init|add|add-key|revoke-key|remove|list|set-relays|show|serve` (signed `registry.json` + `ma-key.hex` in `--dir`); `frxd init --id/--registry/--ma-key`; `frxd key show|rotate`; `member add --previous <old>` for the fallback path. A node with no `[node] relays` discovers them from the registry snapshot (`doc.relays`).
- Aggregate semantics are our implementation choices from a terse spec: requests are `aggregate` envelopes carrying only `period`; replies carry `sent` (broadcasts that month) / `passed` (responses consumed from that member); granularity floor is enforced as YYYY or YYYY-MM only (finer rejected), yearly rolls up months. Revisit with §10 sufficiency review.
## Known gaps (Phase 2/3, intentional — don't fake them)
- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting.
- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting, no SSE streaming (long-poll only), no unicast confidentiality (HPKE).
- Receipts/settlement are out of protocol scope, not unimplemented (I4; Appendix B row 2). Lineage and delegation remain §10 open issues — unspecified without a supply stream, so not buildable as written; don't invent them silently.
- Node query dedup is by `qid` only; replay inside the ±300s skew window remains possible (no nonce cache at nodes), relays have no directory/admission, and there is no end-to-end encryption — relays see everything in clear.
+3 -3
View File
@@ -20,7 +20,7 @@ FRX is a membership federation for retrieval. Content owners answer broadcast qu
3. System Model
Members exchange signed messages via dumb relays (multiply-operated transport, zero protocol authority). Queries are broadcast live to all members; relays hold no history and replay nothing. Responses are unicast from responder to querier. Transport DEFAULT: HTTPS + long-poll/SSE. Bilateral rate limits with visible backpressure are normative; application-layer rejection is silent.
Members exchange signed messages via dumb relays (multiply-operated transport, zero protocol authority). Queries are broadcast live to all members; relays hold no history and replay nothing. Relays MAY flood queries among themselves — copy-only, hop-bounded, duplicate-suppressed — so a member publishes once and coverage scales with relays, not senders. Delivery is per-member and isolated: a lagging member is backpressured visibly and never stalls publishers or peers. Responses are unicast from responder to querier. Transport DEFAULT: HTTPS + long-poll/SSE. Bilateral rate limits with visible backpressure are normative; application-layer rejection is silent.
STREAM
TYPE
@@ -69,7 +69,7 @@ Protocol-silent by design (I2): ranking, ordering, presentation, relevance gatin
6. Membership
The MA governs identity, contract, expulsion — who, never quality. Identifiers are MA-hosted FQDNs (`<label>.frx.<ma-domain>`); no member-controlled DNS is required. Member-hosted identifiers — keys published in the member's own domain and allowlisted by the MA — are planned, not yet normative. The MA maintains a signed, versioned registry snapshot listing identifiers, class, and authorized keys with validity windows. Nodes pin the MA key; the snapshot is the sole authority for the key→identifier binding. Rotation publishes a successor key before retiring its predecessor; revocation removes a key or shortens its validity. Registry outage is fail-static: the last validated snapshot stays in force, and open bootstrap requires an explicit development flag. Admission cost is the Sybil defense. Expulsion grounds: fabrication, admission fraud, sustained abuse — never low quality. Escalation: local throttle → advisory aggregates → MA warning → delisting → expulsion. Aggregates are inadmissible as sanction evidence (I4). Conduct not observable on the wire — link handling, retention, gating — is governed by contract; the protocol neither observes nor adjudicates it. Membership classes: source members (own content) and enrichment members (derived corpora, e.g. GDELT/CC-NEWS bots — metadata-only exposure, transformation logic open and auditable).
The MA governs identity, contract, expulsion — who, never quality. Identifiers are MA-hosted FQDNs (`<label>.frx.<ma-domain>`); no member-controlled DNS is required. Member-hosted identifiers — keys published in the member's own domain and allowlisted by the MA — are planned, not yet normative. The MA maintains a signed, versioned registry snapshot listing identifiers, class, authorized keys with validity windows, and the federation's relay endpoints. Members and queriers may discover relays from it; relays MAY verify sender admission against it, rejecting unlisted keys visibly. Nodes pin the MA key; the snapshot is the sole authority for the key→identifier binding. Rotation publishes a successor key before retiring its predecessor; revocation removes a key or shortens its validity. Registry outage is fail-static: the last validated snapshot stays in force, and open bootstrap requires an explicit development flag. Admission cost is the Sybil defense. Expulsion grounds: fabrication, admission fraud, sustained abuse — never low quality. Escalation: local throttle → advisory aggregates → MA warning → delisting → expulsion. Aggregates are inadmissible as sanction evidence (I4). Conduct not observable on the wire — link handling, retention, gating — is governed by contract; the protocol neither observes nor adjudicates it. Membership classes: source members (own content) and enrichment members (derived corpora, e.g. GDELT/CC-NEWS bots — metadata-only exposure, transformation logic open and auditable).
7. Reference Implementation — frxd
@@ -98,7 +98,7 @@ A conforming implementation: signs all messages with a key authorized for its id
10. Open Issues
Consumer admission tier — automated/invite admission for distributed binaries without weakening the Sybil defense (MA policy, gates §7 adoption).
Relay discovery and default-relay governance (shipped defaults are soft centralization; mitigate with multiple defaults + one-command self-host).
Default-relay governance — registry-listed relays settle discovery, but who operates the defaults remains a soft centralization point (mitigate with multiple operators + one-command self-host).
Delegation grant mechanism.
Member-hosted identifiers — keys published in the member's own DNS instead of the MA registry; MA-hosted is normative until specified.
MA anchor rollover — successor commitment and overlap for the registry signing key.
+13
View File
@@ -277,6 +277,7 @@ pub fn registry_init(dir: &Path) -> Result<()> {
issued_at: now_ts(),
ma_key: String::new(),
members: Vec::new(),
relays: Vec::new(),
};
let signed = registry::sign_registry(doc, &ma);
registry::save_registry(&registry_path, &signed)?;
@@ -390,11 +391,23 @@ pub fn registry_list(dir: &Path) -> Result<()> {
Ok(())
}
pub fn registry_set_relays(dir: &Path, relays: &[String]) -> Result<()> {
mutate_registry(dir, |doc| {
doc.relays = relays.to_vec();
Ok(())
})?;
println!("relays: {}", relays.join(" "));
Ok(())
}
pub fn registry_show(dir: &Path) -> Result<()> {
let (_, signed) = open_registry(dir)?;
println!("ma_key {}", signed.doc.ma_key);
println!("version {}", signed.doc.version);
println!("issued_at {}", signed.doc.issued_at);
for relay in &signed.doc.relays {
println!("relay {relay}");
}
Ok(())
}
+33 -2
View File
@@ -70,6 +70,16 @@ enum Command {
listen: String,
#[arg(long, default_value_t = relay::DEFAULT_CAPACITY)]
capacity: usize,
#[arg(long = "peer")]
peers: Vec<String>,
#[arg(long)]
url: Option<String>,
#[arg(long)]
registry: Option<String>,
#[arg(long)]
ma_key: Option<String>,
#[arg(long, default_value_t = 3)]
max_hops: usize,
},
Status,
Member {
@@ -147,6 +157,10 @@ enum RegistryCommand {
id: String,
},
List,
SetRelays {
#[arg(required = true)]
relays: Vec<String>,
},
Show,
Serve {
#[arg(long, default_value = "127.0.0.1:7800")]
@@ -231,10 +245,26 @@ async fn main() -> Result<()> {
println!("control API POST http://{}/v1/local/query", handle.addr);
tokio::signal::ctrl_c().await?;
}
Command::Relay { listen, capacity } => {
Command::Relay {
listen,
capacity,
peers,
url,
registry,
ma_key,
max_hops,
} => {
let options = relay::RelayOptions {
capacity,
peers,
url,
registry,
ma_key,
max_hops,
};
let (listener, addr) = relay::bind(&listen).await?;
println!("relay listening on http://{addr} (capacity {capacity})");
relay::run(listener, capacity).await?;
relay::run(listener, options).await?;
}
Command::Status => commands::status(&cli.config).await?,
Command::Member { command } => match command {
@@ -271,6 +301,7 @@ async fn main() -> Result<()> {
}
RegistryCommand::Remove { id } => commands::registry_remove(&dir, &id)?,
RegistryCommand::List => commands::registry_list(&dir)?,
RegistryCommand::SetRelays { relays } => commands::registry_set_relays(&dir, &relays)?,
RegistryCommand::Show => commands::registry_show(&dir)?,
RegistryCommand::Serve { listen } => commands::registry_serve(&dir, &listen).await?,
},
+40 -114
View File
@@ -23,7 +23,7 @@ use crate::message::{
AggregateBody, Envelope, QueryBody, ResponseBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY,
TYPE_RESPONSE, build_response, timestamp_is_fresh,
};
use crate::registry::{SignedRegistry, authorized_keys, load_registry, verify_registry};
use crate::registry::Watcher;
#[derive(Default)]
struct Aggregates {
@@ -37,8 +37,7 @@ pub struct Node {
index: LocalIndex,
members: RwLock<Vec<Member>>,
members_mtime: Mutex<Option<SystemTime>>,
registry: Mutex<Option<SignedRegistry>>,
registry_mtime: Mutex<Option<SystemTime>>,
registry: Option<Arc<Watcher>>,
aggregates: Mutex<Aggregates>,
seen: Mutex<HashSet<String>>,
pending: Mutex<HashMap<String, Vec<(String, ResponseBody)>>>,
@@ -122,14 +121,28 @@ impl Node {
.timeout(Duration::from_secs(15))
.build()
.context("building http client")?;
let node = Arc::new(Self {
let registry = match (
config.node.registry.as_deref(),
config.node.ma_key.as_deref(),
) {
(Some(source), Some(ma_key)) => Some(Watcher::new(
source,
ma_key,
Some(config.registry_cache_path()),
)),
(Some(_), None) => return Err(anyhow!("registry configured without ma_key")),
_ => None,
};
if let Some(watcher) = &registry {
watcher.load_initial();
}
Ok(Arc::new(Self {
config,
key,
index,
members: RwLock::new(members),
members_mtime: Mutex::new(members_mtime),
registry: Mutex::new(None),
registry_mtime: Mutex::new(None),
registry,
aggregates: Mutex::new(Aggregates::default()),
seen: Mutex::new(HashSet::new()),
pending: Mutex::new(HashMap::new()),
@@ -137,90 +150,7 @@ impl Node {
client,
sent: AtomicU64::new(0),
received: AtomicU64::new(0),
});
node.load_initial_registry();
Ok(node)
}
fn load_initial_registry(&self) {
let Some(source) = &self.config.node.registry else {
return;
};
let result = if source.starts_with("http://") || source.starts_with("https://") {
load_registry(&self.config.registry_cache_path())
} else {
let path = std::path::PathBuf::from(source);
let loaded = load_registry(&path);
if loaded.is_ok() {
let mtime = fs::metadata(&path)
.and_then(|metadata| metadata.modified())
.ok();
*self.registry_mtime.lock().expect("registry mtime lock") = mtime;
}
loaded
};
if let Ok(signed) = result {
if let Err(error) = self.verify_and_apply_registry(signed) {
eprintln!("cached registry rejected: {error}");
}
}
}
fn verify_and_apply_registry(&self, signed: SignedRegistry) -> Result<()> {
let ma_key = self
.config
.node
.ma_key
.as_deref()
.ok_or_else(|| anyhow!("registry configured without ma_key"))?;
verify_registry(&signed, ma_key)?;
{
let current = self.registry.lock().expect("registry lock");
if let Some(existing) = current.as_ref() {
if signed.doc.version <= existing.doc.version {
return Err(anyhow!("registry version rollback rejected"));
}
}
}
*self.registry.lock().expect("registry lock") = Some(signed);
Ok(())
}
fn refresh_registry(&self) {
let Some(source) = &self.config.node.registry else {
return;
};
if source.starts_with("http://") || source.starts_with("https://") {
return;
}
let path = std::path::PathBuf::from(source);
let mtime = fs::metadata(&path)
.and_then(|metadata| metadata.modified())
.ok();
{
let last = self.registry_mtime.lock().expect("registry mtime lock");
if *last == mtime {
return;
}
}
let Ok(signed) = load_registry(&path) else {
return;
};
*self.registry_mtime.lock().expect("registry mtime lock") = mtime;
if let Err(error) = self.verify_and_apply_registry(signed) {
eprintln!("registry update rejected: {error}");
}
}
async fn fetch_registry(&self) -> Result<()> {
let Some(url) = &self.config.node.registry else {
return Ok(());
};
let response = self.client.get(url).send().await?;
let signed: SignedRegistry = response.json().await?;
self.verify_and_apply_registry(signed.clone())?;
crate::registry::save_registry(&self.config.registry_cache_path(), &signed)?;
Ok(())
}))
}
fn refresh_members(&self) {
@@ -311,23 +241,31 @@ impl Node {
.with_context(|| format!("binding {}", node.config.node.listen))?;
let addr = listener.local_addr()?;
let mut tasks = Vec::new();
if let Some(source) = &node.config.node.registry {
if source.starts_with("http://") || source.starts_with("https://") {
if let Err(error) = node.fetch_registry().await {
if let Some(watcher) = node.registry.clone() {
if watcher.is_url() {
if let Err(error) = watcher.fetch(&node.client).await {
eprintln!("initial registry fetch failed: {error}");
}
let node_for_registry = node.clone();
let client = node.client.clone();
tasks.push(tokio::spawn(async move {
loop {
if let Err(error) = node_for_registry.fetch_registry().await {
tokio::time::sleep(Duration::from_secs(60)).await;
if let Err(error) = watcher.fetch(&client).await {
eprintln!("registry fetch failed: {error}");
}
tokio::time::sleep(Duration::from_secs(60)).await;
}
}));
}
}
for relay in node.config.node.relays.clone() {
let relays = if node.config.node.relays.is_empty() {
node.registry
.as_ref()
.map(|watcher| watcher.relays())
.unwrap_or_default()
} else {
node.config.node.relays.clone()
};
for relay in relays {
tasks.push(tokio::spawn(poll_relay(node.clone(), relay)));
}
let app = router(node.clone());
@@ -456,18 +394,11 @@ impl Node {
return;
}
self.refresh_members();
self.refresh_registry();
let (class, listed) = if self.config.node.registry.is_some() {
let registry = self.registry.lock().expect("registry lock");
let authorized = registry
.as_ref()
.map(|signed| authorized_keys(signed, now_ts()));
match authorized {
Some(map) => match map.get(&envelope.key) {
Some((id, class)) if id == &envelope.from => (Some(class.clone()), true),
let (class, listed) = if let Some(watcher) = &self.registry {
watcher.refresh_if_changed();
match watcher.authorized(&envelope.key, now_ts()) {
Some((id, class)) if id == envelope.from => (Some(class), true),
_ => (None, false),
},
None => (None, false),
}
} else {
let members = self.members.read().expect("members lock");
@@ -761,12 +692,7 @@ async fn local_query(
async fn local_status(State(node): State<Arc<Node>>) -> Response {
let aggregates = node.aggregate_for(&current_period(), None);
let registry_version = node
.registry
.lock()
.expect("registry lock")
.as_ref()
.map(|signed| signed.doc.version);
let registry_version = node.registry.as_ref().and_then(|watcher| watcher.version());
Json(json!({
"name": node.config.node.name,
"id": node.config.node.id,
+127 -1
View File
@@ -1,6 +1,8 @@
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use std::time::SystemTime;
use anyhow::{Context, Result, anyhow};
use serde::{Deserialize, Serialize};
@@ -35,6 +37,8 @@ pub struct RegistryDoc {
pub ma_key: String,
#[serde(default)]
pub members: Vec<RegistryMember>,
#[serde(default)]
pub relays: Vec<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -75,6 +79,127 @@ pub fn save_registry(path: &Path, signed: &SignedRegistry) -> Result<()> {
Ok(())
}
pub struct Watcher {
source: String,
ma_key: String,
cache_path: Option<PathBuf>,
current: Mutex<Option<SignedRegistry>>,
mtime: Mutex<Option<SystemTime>>,
}
impl Watcher {
pub fn new(source: &str, ma_key: &str, cache_path: Option<PathBuf>) -> Arc<Self> {
Arc::new(Self {
source: source.to_string(),
ma_key: ma_key.to_string(),
cache_path,
current: Mutex::new(None),
mtime: Mutex::new(None),
})
}
pub fn is_url(&self) -> bool {
self.source.starts_with("http://") || self.source.starts_with("https://")
}
pub fn load_initial(&self) {
let loaded = if self.is_url() {
match &self.cache_path {
Some(cache) => load_registry(cache),
None => Err(anyhow!("no cache path for URL registry")),
}
} else {
let path = PathBuf::from(&self.source);
let loaded = load_registry(&path);
if loaded.is_ok() {
*self.mtime.lock().expect("registry mtime lock") = fs::metadata(&path)
.and_then(|metadata| metadata.modified())
.ok();
}
loaded
};
if let Ok(signed) = loaded {
if let Err(error) = self.apply(signed) {
eprintln!("cached registry rejected: {error}");
}
}
}
pub fn refresh_if_changed(&self) {
if self.is_url() {
return;
}
let path = PathBuf::from(&self.source);
let mtime = fs::metadata(&path)
.and_then(|metadata| metadata.modified())
.ok();
{
let last = self.mtime.lock().expect("registry mtime lock");
if *last == mtime {
return;
}
}
let Ok(signed) = load_registry(&path) else {
return;
};
*self.mtime.lock().expect("registry mtime lock") = mtime;
if let Err(error) = self.apply(signed) {
eprintln!("registry update rejected: {error}");
}
}
pub async fn fetch(&self, client: &reqwest::Client) -> Result<()> {
if !self.is_url() {
return Ok(());
}
let response = client.get(&self.source).send().await?;
let signed: SignedRegistry = response.json().await?;
self.apply(signed.clone())?;
if let Some(cache) = &self.cache_path {
save_registry(cache, &signed)?;
}
Ok(())
}
fn apply(&self, signed: SignedRegistry) -> Result<()> {
verify_registry(&signed, &self.ma_key)?;
{
let current = self.current.lock().expect("registry lock");
if let Some(existing) = current.as_ref() {
if signed.doc.version <= existing.doc.version {
return Err(anyhow!("registry version rollback rejected"));
}
}
}
*self.current.lock().expect("registry lock") = Some(signed);
Ok(())
}
pub fn authorized(&self, key: &str, now: u64) -> Option<(String, String)> {
let current = self.current.lock().expect("registry lock");
current
.as_ref()
.and_then(|signed| authorized_keys(signed, now).remove(key))
}
pub fn version(&self) -> Option<u64> {
self.current
.lock()
.expect("registry lock")
.as_ref()
.map(|signed| signed.doc.version)
}
pub fn relays(&self) -> Vec<String> {
self.current
.lock()
.expect("registry lock")
.as_ref()
.map(|signed| signed.doc.relays.clone())
.unwrap_or_default()
}
}
pub fn authorized_keys(signed: &SignedRegistry, now: u64) -> HashMap<String, (String, String)> {
let mut authorized = HashMap::new();
for member in &signed.doc.members {
@@ -117,6 +242,7 @@ mod tests {
issued_at: now_ts(),
ma_key: String::new(),
members,
relays: Vec::new(),
},
key,
)
+220 -29
View File
@@ -3,7 +3,7 @@ use std::net::SocketAddr;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use anyhow::Result;
use anyhow::{Result, anyhow};
use axum::extract::{Query, State};
use axum::http::{StatusCode, header};
use axum::response::{IntoResponse, Response};
@@ -15,13 +15,40 @@ use tokio::net::TcpListener;
use crate::crypto::{now_ts, poll_signing_bytes, random_nonce, verify_signature};
use crate::message::{Envelope, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE, timestamp_is_fresh};
use crate::registry::Watcher;
pub const DEFAULT_CAPACITY: usize = 256;
const CHALLENGE_TTL: Duration = Duration::from_secs(120);
const SEEN_TTL: Duration = Duration::from_secs(30);
#[derive(Clone, Debug)]
pub struct RelayOptions {
pub capacity: usize,
pub peers: Vec<String>,
pub url: Option<String>,
pub registry: Option<String>,
pub ma_key: Option<String>,
pub max_hops: usize,
}
impl Default for RelayOptions {
fn default() -> Self {
Self {
capacity: DEFAULT_CAPACITY,
peers: Vec::new(),
url: None,
registry: None,
ma_key: None,
max_hops: 3,
}
}
}
#[derive(Default)]
struct MemberQueue {
items: VecDeque<(u64, Envelope)>,
lagging: bool,
missed: u64,
}
struct Challenge {
@@ -32,57 +59,136 @@ struct Challenge {
struct Inner {
members: HashMap<String, MemberQueue>,
challenges: HashMap<String, Challenge>,
seen: HashMap<String, Instant>,
seq: u64,
}
pub struct Relay {
inner: Mutex<Inner>,
capacity: usize,
options: RelayOptions,
registry: Option<Arc<Watcher>>,
client: reqwest::Client,
}
impl Relay {
pub fn new(capacity: usize) -> Arc<Self> {
Arc::new(Self {
pub fn new(options: RelayOptions) -> Result<Arc<Self>> {
let registry = match (options.registry.as_deref(), options.ma_key.as_deref()) {
(Some(source), Some(ma_key)) => Some(Watcher::new(source, ma_key, None)),
(Some(_), None) => return Err(anyhow!("registry configured without ma_key")),
_ => None,
};
if !options.peers.is_empty() && options.url.is_none() {
return Err(anyhow!("--url is required when --peer is set"));
}
let relay = Arc::new(Self {
inner: Mutex::new(Inner {
members: HashMap::new(),
challenges: HashMap::new(),
seen: HashMap::new(),
seq: 0,
}),
capacity,
})
options,
registry,
client: reqwest::Client::builder()
.timeout(Duration::from_secs(5))
.build()?,
});
if let Some(watcher) = &relay.registry {
watcher.load_initial();
}
Ok(relay)
}
fn push(&self, targets: Option<&str>, envelope: Envelope) -> Result<usize, StatusCode> {
let mut inner = self.inner.lock().expect("relay lock");
if inner
.members
.values()
.any(|q| q.items.len() >= self.capacity)
{
return Err(StatusCode::TOO_MANY_REQUESTS);
fn admit(&self, envelope: &Envelope) -> bool {
match &self.registry {
Some(watcher) => {
watcher.refresh_if_changed();
watcher.authorized(&envelope.key, now_ts()).is_some()
}
None => true,
}
}
fn mark_seen(&self, signature: &str) -> bool {
let mut inner = self.inner.lock().expect("relay lock");
let now = Instant::now();
inner
.seen
.retain(|_, at| now.duration_since(*at) < SEEN_TTL);
if inner.seen.contains_key(signature) {
return false;
}
inner.seen.insert(signature.to_string(), now);
true
}
fn push_local(&self, target: Option<&str>, envelope: Envelope) -> Result<usize, StatusCode> {
let mut inner = self.inner.lock().expect("relay lock");
inner.seq += 1;
let seq = inner.seq;
match targets {
match target {
Some(member) => {
let Some(queue) = inner.members.get_mut(member) else {
return Err(StatusCode::NOT_FOUND);
};
if queue.items.len() >= self.options.capacity {
queue.lagging = true;
queue.missed += 1;
return Err(StatusCode::TOO_MANY_REQUESTS);
}
queue.items.push_back((seq, envelope));
Ok(1)
}
None => {
let from = envelope.key.clone();
inner.members.entry(from).or_default();
let publisher = envelope.key.clone();
inner.members.entry(publisher).or_default();
let mut delivered = 0;
for queue in inner.members.values_mut() {
if queue.items.len() >= self.options.capacity {
queue.lagging = true;
queue.missed += 1;
} else {
queue.items.push_back((seq, envelope.clone()));
delivered += 1;
}
}
Ok(delivered)
}
}
}
fn forward(&self, envelope: Envelope, origin: Option<&str>, hops: usize) {
if self.options.peers.is_empty() {
return;
}
let Some(url) = self.options.url.clone() else {
return;
};
let peers: Vec<String> = self
.options
.peers
.iter()
.filter(|peer| Some(peer.as_str()) != origin)
.cloned()
.collect();
if peers.is_empty() {
return;
}
let client = self.client.clone();
tokio::spawn(async move {
for peer in peers {
let target = format!(
"{}/v1/federation?origin={}&hops={}",
peer.trim_end_matches('/'),
url,
hops
);
if let Err(error) = client.post(&target).json(&envelope).send().await {
eprintln!("federation forward to {peer} failed: {error}");
}
}
});
}
}
pub fn router(relay: Arc<Relay>) -> Router {
@@ -90,13 +196,30 @@ pub fn router(relay: Arc<Relay>) -> Router {
.route("/health", get(health))
.route("/v1/challenge", get(challenge))
.route("/v1/publish", post(publish))
.route("/v1/federation", post(federation))
.route("/v1/unicast", post(unicast))
.route("/v1/poll", get(poll))
.with_state(relay)
}
pub async fn run(listener: TcpListener, capacity: usize) -> Result<()> {
let relay = Relay::new(capacity);
pub async fn run(listener: TcpListener, options: RelayOptions) -> Result<()> {
let relay = Relay::new(options)?;
if let Some(watcher) = relay.registry.clone() {
if watcher.is_url() {
let client = relay.client.clone();
if let Err(error) = watcher.fetch(&client).await {
eprintln!("initial registry fetch failed: {error}");
}
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(60)).await;
if let Err(error) = watcher.fetch(&client).await {
eprintln!("registry fetch failed: {error}");
}
}
});
}
}
axum::serve(listener, router(relay)).await?;
Ok(())
}
@@ -152,14 +275,65 @@ async fn publish(State(relay): State<Arc<Relay>>, Json(envelope): Json<Envelope>
if envelope.msg_type != TYPE_QUERY {
return bad_request("relay carries broadcast queries only");
}
match relay.push(None, envelope) {
Ok(delivered) => (
if !relay.admit(&envelope) {
return bad_request("sender not admitted");
}
relay.mark_seen(&envelope.sig);
let delivered = relay.push_local(None, envelope.clone()).unwrap_or(0);
relay.forward(envelope, None, 1);
(
StatusCode::ACCEPTED,
Json(json!({ "delivered": delivered })),
)
.into_response(),
Err(status) => backpressure(status),
.into_response()
}
#[derive(Deserialize)]
struct FederationParams {
origin: Option<String>,
hops: Option<usize>,
}
async fn federation(
State(relay): State<Arc<Relay>>,
Query(params): Query<FederationParams>,
Json(envelope): Json<Envelope>,
) -> Response {
let Some(origin) = params.origin.clone() else {
return bad_request("origin required");
};
if !relay.options.peers.iter().any(|peer| peer == &origin) {
return bad_request("unknown peer origin");
}
if envelope.verify().is_err() {
return bad_request("invalid signature");
}
if !timestamp_is_fresh(envelope.ts, now_ts()) {
return bad_request("stale timestamp");
}
if envelope.msg_type != TYPE_QUERY {
return bad_request("relay carries broadcast queries only");
}
if !relay.admit(&envelope) {
return bad_request("sender not admitted");
}
if !relay.mark_seen(&envelope.sig) {
return (
StatusCode::ACCEPTED,
Json(json!({ "delivered": 0, "duplicate": true })),
)
.into_response();
}
let delivered = relay.push_local(None, envelope.clone()).unwrap_or(0);
let hops = params.hops.unwrap_or(1);
if hops < relay.options.max_hops {
relay.forward(envelope, Some(&origin), hops + 1);
}
(
StatusCode::ACCEPTED,
Json(json!({ "delivered": delivered })),
)
.into_response()
}
#[derive(Deserialize)]
@@ -181,14 +355,17 @@ async fn unicast(
if envelope.msg_type != TYPE_RESPONSE && envelope.msg_type != TYPE_AGGREGATE {
return bad_request("unicast carries responses and aggregates only");
}
match relay.push(Some(&params.to), envelope) {
if !relay.admit(&envelope) {
return bad_request("sender not admitted");
}
match relay.push_local(Some(&params.to), envelope) {
Ok(_) => (StatusCode::OK, Json(json!({ "delivered": true }))).into_response(),
Err(StatusCode::NOT_FOUND) => (
StatusCode::NOT_FOUND,
Json(json!({ "error": "member not connected" })),
)
.into_response(),
Err(status) => backpressure(status),
Err(status) => backpressure(status, 0),
}
}
@@ -225,15 +402,29 @@ async fn poll(State(relay): State<Arc<Relay>>, Query(params): Query<PollParams>)
let timeout = Duration::from_millis(params.timeout_ms.unwrap_or(25_000).min(60_000));
let deadline = Instant::now() + timeout;
loop {
let batch: Vec<Envelope> = {
let (batch, lagged): (Vec<Envelope>, Option<u64>) = {
let mut inner = relay.inner.lock().expect("relay lock");
let queue = inner.members.entry(params.member.clone()).or_default();
if queue.lagging {
let missed = queue.missed;
queue.lagging = false;
queue.missed = 0;
queue.items.clear();
(Vec::new(), Some(missed))
} else {
(
queue
.items
.drain(..)
.map(|(_, envelope)| envelope)
.collect()
.collect(),
None,
)
}
};
if let Some(missed) = lagged {
return backpressure(StatusCode::TOO_MANY_REQUESTS, missed);
}
if !batch.is_empty() {
return (StatusCode::OK, Json(json!({ "messages": batch }))).into_response();
}
@@ -252,11 +443,11 @@ fn unauthorized(message: &str) -> Response {
(StatusCode::UNAUTHORIZED, Json(json!({ "error": message }))).into_response()
}
fn backpressure(status: StatusCode) -> Response {
fn backpressure(status: StatusCode, missed: u64) -> Response {
(
status,
[(header::RETRY_AFTER, "1")],
Json(json!({ "error": "transport backpressure" })),
Json(json!({ "error": "lagging", "missed": missed })),
)
.into_response()
}
+5 -1
View File
@@ -49,7 +49,11 @@ pub async fn spawn_relay() -> String {
pub async fn spawn_relay_with_capacity(capacity: usize) -> String {
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
tokio::spawn(async move {
let _ = relay::run(listener, capacity).await;
let options = relay::RelayOptions {
capacity,
..Default::default()
};
let _ = relay::run(listener, options).await;
});
format!("http://{addr}")
}
+52 -13
View File
@@ -478,38 +478,53 @@ async fn stale_envelopes_are_rejected() {
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn backpressure_is_visible_and_recoverable() {
async fn lagging_member_is_isolated_and_reported() {
let relay_url = spawn_relay_with_capacity(1).await;
let http = client();
let member = Keypair::generate();
let active = Keypair::generate();
let slow = Keypair::generate();
let publisher = Keypair::generate();
register(&http, &relay_url, &member).await;
register(&http, &relay_url, &active).await;
register(&http, &relay_url, &slow).await;
let first = query_envelope(&publisher, &publisher.public_hex(), "one", 5);
assert_eq!(
publish(&http, &relay_url, &first).await.status(),
reqwest::StatusCode::ACCEPTED
);
assert_eq!(
poll_messages(&http, &relay_url, &active, 300).await.len(),
1
);
let second = query_envelope(&publisher, &publisher.public_hex(), "two", 5);
let response = publish(&http, &relay_url, &second).await;
assert_eq!(
publish(&http, &relay_url, &second).await.status(),
reqwest::StatusCode::ACCEPTED,
"a lagging subscriber must not stall publishers"
);
assert_eq!(
poll_messages(&http, &relay_url, &active, 300).await.len(),
1
);
let response = poll(&http, &relay_url, &slow, 300).await;
assert_eq!(response.status(), reqwest::StatusCode::TOO_MANY_REQUESTS);
assert_eq!(
response
.headers()
.get("retry-after")
.and_then(|v| v.to_str().ok()),
.and_then(|value| value.to_str().ok()),
Some("1")
);
assert_eq!(
poll_messages(&http, &relay_url, &member, 300).await.len(),
1
let payload: Value = response.json().await.unwrap();
assert!(
payload.get("missed").and_then(Value::as_u64).unwrap_or(0) >= 1,
"lagging member must be told how much it missed: {payload}"
);
assert_eq!(
poll_messages(&http, &relay_url, &publisher, 300)
.await
.len(),
1
poll(&http, &relay_url, &slow, 50).await.status(),
reqwest::StatusCode::NO_CONTENT
);
let third = query_envelope(&publisher, &publisher.public_hex(), "three", 5);
@@ -517,6 +532,30 @@ async fn backpressure_is_visible_and_recoverable() {
publish(&http, &relay_url, &third).await.status(),
reqwest::StatusCode::ACCEPTED
);
assert_eq!(poll_messages(&http, &relay_url, &slow, 300).await.len(), 1);
let fourth = query_envelope(&publisher, &publisher.public_hex(), "four", 5);
assert_eq!(
publish(&http, &relay_url, &fourth).await.status(),
reqwest::StatusCode::ACCEPTED
);
let fifth = query_envelope(&publisher, &publisher.public_hex(), "five", 5);
assert_eq!(
publish(&http, &relay_url, &fifth).await.status(),
reqwest::StatusCode::ACCEPTED
);
let direct = test_envelope(
&publisher,
TYPE_RESPONSE,
json!({"qid": "q", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
);
assert_eq!(
unicast(&http, &relay_url, &slow.public_hex(), &direct)
.await
.status(),
reqwest::StatusCode::TOO_MANY_REQUESTS,
"unicast to a lagging member must be visibly rejected"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+251
View File
@@ -0,0 +1,251 @@
mod common;
use std::fs;
use std::path::Path;
use std::time::Duration;
use common::{
client, collection, config_for, messages_of_type, poll, poll_messages, publish, query_envelope,
register, test_envelope,
};
use frxd::crypto::{Keypair, now_ts};
use frxd::index::LocalIndex;
use frxd::message::{EXPOSURE_FULL, TYPE_QUERY, TYPE_RESPONSE};
use frxd::node::Node;
use frxd::registry::{self, KeyEntry, RegistryDoc, RegistryMember, SignedRegistry};
use frxd::relay::{self, RelayOptions};
async fn spawn_relay_with(options: RelayOptions) -> String {
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
tokio::spawn(async move {
let _ = relay::run(listener, options).await;
});
format!("http://{addr}")
}
async fn federation_mesh() -> [String; 3] {
let mut listeners = Vec::new();
let mut urls = Vec::new();
for _ in 0..3 {
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
listeners.push(listener);
urls.push(format!("http://{addr}"));
}
let peers: Vec<Vec<String>> = vec![
vec![urls[1].clone(), urls[2].clone()],
vec![urls[0].clone(), urls[2].clone()],
vec![urls[0].clone(), urls[1].clone()],
];
for (index, listener) in listeners.into_iter().enumerate() {
let options = RelayOptions {
url: Some(urls[index].clone()),
peers: peers[index].clone(),
..Default::default()
};
tokio::spawn(async move {
let _ = relay::run(listener, options).await;
});
}
[urls[0].clone(), urls[1].clone(), urls[2].clone()]
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn mesh_floods_queries_exactly_once() {
let [r1, _r2, r3] = federation_mesh().await;
let http = client();
let member = Keypair::generate();
let publisher = Keypair::generate();
register(&http, &r3, &member).await;
let envelope = query_envelope(&publisher, &publisher.public_hex(), "mesh", 5);
assert!(publish(&http, &r1, &envelope).await.status().is_success());
let messages = poll_messages(&http, &r3, &member, 900).await;
assert_eq!(messages.len(), 1, "expected one flooded copy: {messages:?}");
tokio::time::sleep(Duration::from_millis(300)).await;
let duplicates = poll_messages(&http, &r3, &member, 200).await;
assert!(
duplicates.is_empty(),
"mesh delivered duplicates: {duplicates:?}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn federation_rejects_unknown_origin() {
let [r1, _r2, _r3] = federation_mesh().await;
let http = client();
let key = Keypair::generate();
let envelope = query_envelope(&key, &key.public_hex(), "spoof", 5);
let response = http
.post(format!(
"{r1}/v1/federation?origin=http://127.0.0.1:1&hops=1"
))
.json(&envelope)
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
}
fn registry_doc(
ma: &Keypair,
members: Vec<RegistryMember>,
relays: Vec<String>,
version: u64,
) -> SignedRegistry {
registry::sign_registry(
RegistryDoc {
version,
issued_at: now_ts(),
ma_key: String::new(),
members,
relays,
},
ma,
)
}
fn member(id: &str, key: &Keypair, class: &str) -> RegistryMember {
RegistryMember {
id: id.to_string(),
class: class.to_string(),
keys: vec![KeyEntry {
key: key.public_hex(),
not_before: 0,
not_after: None,
}],
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn relay_admission_gates_unlisted_keys() {
let root = tempfile::tempdir().unwrap();
let ma = Keypair::generate();
let alice = Keypair::generate();
let registry_path = root.path().join("registry.json");
registry::save_registry(
&registry_path,
&registry_doc(
&ma,
vec![member("alice.frx.example", &alice, "source")],
vec![],
1,
),
)
.unwrap();
let relay_url = spawn_relay_with(RelayOptions {
registry: Some(registry_path.display().to_string()),
ma_key: Some(ma.public_hex()),
..Default::default()
})
.await;
let http = client();
let accepted = query_envelope(&alice, "alice.frx.example", "rust", 5);
assert_eq!(
publish(&http, &relay_url, &accepted).await.status(),
reqwest::StatusCode::ACCEPTED
);
let stranger = Keypair::generate();
let denied = query_envelope(&stranger, &stranger.public_hex(), "rust", 5);
assert_eq!(
publish(&http, &relay_url, &denied).await.status(),
reqwest::StatusCode::BAD_REQUEST,
"unlisted key must not publish"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn node_discovers_relays_from_registry() {
let root = tempfile::tempdir().unwrap();
let ma = Keypair::generate();
let alice = Keypair::generate();
let (listener, relay_addr) = relay::bind("127.0.0.1:0").await.unwrap();
tokio::spawn(async move {
let _ = relay::run(listener, RelayOptions::default()).await;
});
let relay_url = format!("http://{relay_addr}");
let docs = root.path().join("bob-docs");
fs::create_dir_all(&docs).unwrap();
fs::write(docs.join("doc.txt"), "registry discovery rust document").unwrap();
let mut bob_config = config_for(&root.path().join("bob"), "bob", "http://127.0.0.1:1");
bob_config.node.relays = Vec::new();
bob_config.node.id = Some("bob.frx.example".to_string());
let registry_path = root.path().join("registry.json");
bob_config.node.registry = Some(registry_path.display().to_string());
bob_config.node.ma_key = Some(ma.public_hex());
bob_config.node.dev_bootstrap = false;
registry::save_registry(
&registry_path,
&registry_doc(
&ma,
vec![
member("alice.frx.example", &alice, "source"),
member("bob.frx.example", &bob_config.load_key().unwrap(), "source"),
],
vec![relay_url.clone()],
1,
),
)
.unwrap();
{
let index = LocalIndex::open(&bob_config.index_dir()).unwrap();
index
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
.unwrap();
}
let _bob = Node::start(bob_config).await.unwrap();
let http = client();
register(&http, &relay_url, &alice).await;
let envelope = query_envelope(&alice, "alice.frx.example", "rust", 5);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let mut responses = Vec::new();
for _ in 0..5 {
let messages = poll_messages(&http, &relay_url, &alice, 200).await;
responses.extend(messages_of_type(&messages, TYPE_RESPONSE));
if !responses.is_empty() {
break;
}
}
assert_eq!(
responses.len(),
1,
"node did not discover its relay from the registry"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn unknown_types_never_flood() {
let [r1, _r2, _r3] = federation_mesh().await;
let http = client();
let key = Keypair::generate();
let response = test_envelope(
&key,
TYPE_RESPONSE,
serde_json::json!({"qid": "q", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
);
let result = http
.post(format!(
"{r1}/v1/federation?origin={}&hops=1",
"http://unknown"
))
.json(&response)
.send()
.await
.unwrap();
assert_eq!(result.status(), reqwest::StatusCode::BAD_REQUEST);
let _ = TYPE_QUERY;
}
+1
View File
@@ -41,6 +41,7 @@ fn ma_registry(ma: &Keypair, members: Vec<RegistryMember>, version: u64) -> Sign
issued_at: now_ts(),
ma_key: String::new(),
members,
relays: Vec::new(),
},
ma,
)