Federated relay tier: peer flooding, per-member isolation, registry admission and relay discovery
This commit is contained in:
@@ -3,7 +3,7 @@
|
||||
## Repo shape
|
||||
- `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
|
||||
- `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
|
||||
- Commands: `cargo build`, `cargo test` (80 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
|
||||
- Commands: `cargo build`, `cargo test` (85 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
|
||||
- E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.
|
||||
|
||||
## Editing the spec
|
||||
@@ -21,7 +21,7 @@
|
||||
- Relay addresses transport mailboxes by `key` (unicast `to` = recipient pubkey; queues keyed by pubkey); the identifier is protocol identity only. Registry binding checks `map[key].id == from`.
|
||||
- Built: envelope/query/response, Tantivy index, aggregates, member directory. Not built: dashboard UI, directory watching, TLS, lineage/delegation. Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
|
||||
- Economics is out of protocol scope (I4: aggregates advise, contracts govern): no receipt, citation, pricing, or settlement fields or message types exist or may be added.
|
||||
- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queue drained on poll), and returns 429 + Retry-After under backpressure — never silent drops. Mailbox polls require proof of key possession: `GET /v1/challenge` then a signed single-use nonce, so knowing a pubkey is not enough to drain its queue.
|
||||
- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queues drained on poll), and requires challenge–response proof of key possession for mailbox polls. Per-member queues are isolated: a lagging member gets 429 + Retry-After with a `missed` count; publishers and other members are never stalled. Relays MAY flood to configured peers (`/v1/federation`, `--peer`+`--url`, hop-bounded, seen-set dedup without suppressing identical direct publishes) and MAY gate senders against a registry (`--registry`+`--ma-key`).
|
||||
- Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits results with honest `truncated`/`more_available` and no scores (I6). BM25 order is a local implementation detail, not protocol surface.
|
||||
- Index layout: Tantivy at `<data_dir>/index`, collections manifest at `<data_dir>/collections.toml`; `exposure` (metadata|full) gates whether `content` is returned.
|
||||
- Egress checks live in the responder path (`src/node.rs` `respond`), not the relay — keep private collections unreachable there.
|
||||
@@ -30,11 +30,11 @@
|
||||
- Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale).
|
||||
- Member authority (Draft 0.5 §6): the MA-signed registry snapshot is authoritative when configured (`[node] registry` = file path or URL, `ma_key` pinned; monotonic version — rollback and forgery close the node; file path is mtime-reloaded, URL is fetched at start + every 60s and cached to `<data_dir>/registry-cache.json`, so outage fails static). Keys carry optional validity windows (`not_before`/`not_after`); rotation = `registry add-key` then `revoke-key`.
|
||||
- `<data_dir>/members.toml` (name, pubkey, class, `previous` keys, mtime-reloaded) is a dev/local fallback used only when no registry is configured; empty directory without a registry is open bootstrap only when `dev_bootstrap = true` (RFC §6: explicit dev flag). Receivers drop content-bearing responses from enrichment-class senders (metadata-only, §6).
|
||||
- MA tooling: `frxd registry init|add|add-key|revoke-key|remove|list|show|serve` (signed `registry.json` + `ma-key.hex` in `--dir`); `frxd init --id/--registry/--ma-key`; `frxd key show|rotate`; `member add --previous <old>` for the fallback path.
|
||||
- MA tooling: `frxd registry init|add|add-key|revoke-key|remove|list|set-relays|show|serve` (signed `registry.json` + `ma-key.hex` in `--dir`); `frxd init --id/--registry/--ma-key`; `frxd key show|rotate`; `member add --previous <old>` for the fallback path. A node with no `[node] relays` discovers them from the registry snapshot (`doc.relays`).
|
||||
- Aggregate semantics are our implementation choices from a terse spec: requests are `aggregate` envelopes carrying only `period`; replies carry `sent` (broadcasts that month) / `passed` (responses consumed from that member); granularity floor is enforced as YYYY or YYYY-MM only (finer rejected), yearly rolls up months. Revisit with §10 sufficiency review.
|
||||
|
||||
## Known gaps (Phase 2/3, intentional — don't fake them)
|
||||
- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting.
|
||||
- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting, no SSE streaming (long-poll only), no unicast confidentiality (HPKE).
|
||||
- Receipts/settlement are out of protocol scope, not unimplemented (I4; Appendix B row 2). Lineage and delegation remain §10 open issues — unspecified without a supply stream, so not buildable as written; don't invent them silently.
|
||||
- Node query dedup is by `qid` only; replay inside the ±300s skew window remains possible (no nonce cache at nodes), relays have no directory/admission, and there is no end-to-end encryption — relays see everything in clear.
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ FRX is a membership federation for retrieval. Content owners answer broadcast qu
|
||||
|
||||
3. System Model
|
||||
|
||||
Members exchange signed messages via dumb relays (multiply-operated transport, zero protocol authority). Queries are broadcast live to all members; relays hold no history and replay nothing. Responses are unicast from responder to querier. Transport DEFAULT: HTTPS + long-poll/SSE. Bilateral rate limits with visible backpressure are normative; application-layer rejection is silent.
|
||||
Members exchange signed messages via dumb relays (multiply-operated transport, zero protocol authority). Queries are broadcast live to all members; relays hold no history and replay nothing. Relays MAY flood queries among themselves — copy-only, hop-bounded, duplicate-suppressed — so a member publishes once and coverage scales with relays, not senders. Delivery is per-member and isolated: a lagging member is backpressured visibly and never stalls publishers or peers. Responses are unicast from responder to querier. Transport DEFAULT: HTTPS + long-poll/SSE. Bilateral rate limits with visible backpressure are normative; application-layer rejection is silent.
|
||||
|
||||
STREAM
|
||||
TYPE
|
||||
@@ -69,7 +69,7 @@ Protocol-silent by design (I2): ranking, ordering, presentation, relevance gatin
|
||||
|
||||
6. Membership
|
||||
|
||||
The MA governs identity, contract, expulsion — who, never quality. Identifiers are MA-hosted FQDNs (`<label>.frx.<ma-domain>`); no member-controlled DNS is required. Member-hosted identifiers — keys published in the member's own domain and allowlisted by the MA — are planned, not yet normative. The MA maintains a signed, versioned registry snapshot listing identifiers, class, and authorized keys with validity windows. Nodes pin the MA key; the snapshot is the sole authority for the key→identifier binding. Rotation publishes a successor key before retiring its predecessor; revocation removes a key or shortens its validity. Registry outage is fail-static: the last validated snapshot stays in force, and open bootstrap requires an explicit development flag. Admission cost is the Sybil defense. Expulsion grounds: fabrication, admission fraud, sustained abuse — never low quality. Escalation: local throttle → advisory aggregates → MA warning → delisting → expulsion. Aggregates are inadmissible as sanction evidence (I4). Conduct not observable on the wire — link handling, retention, gating — is governed by contract; the protocol neither observes nor adjudicates it. Membership classes: source members (own content) and enrichment members (derived corpora, e.g. GDELT/CC-NEWS bots — metadata-only exposure, transformation logic open and auditable).
|
||||
The MA governs identity, contract, expulsion — who, never quality. Identifiers are MA-hosted FQDNs (`<label>.frx.<ma-domain>`); no member-controlled DNS is required. Member-hosted identifiers — keys published in the member's own domain and allowlisted by the MA — are planned, not yet normative. The MA maintains a signed, versioned registry snapshot listing identifiers, class, authorized keys with validity windows, and the federation's relay endpoints. Members and queriers may discover relays from it; relays MAY verify sender admission against it, rejecting unlisted keys visibly. Nodes pin the MA key; the snapshot is the sole authority for the key→identifier binding. Rotation publishes a successor key before retiring its predecessor; revocation removes a key or shortens its validity. Registry outage is fail-static: the last validated snapshot stays in force, and open bootstrap requires an explicit development flag. Admission cost is the Sybil defense. Expulsion grounds: fabrication, admission fraud, sustained abuse — never low quality. Escalation: local throttle → advisory aggregates → MA warning → delisting → expulsion. Aggregates are inadmissible as sanction evidence (I4). Conduct not observable on the wire — link handling, retention, gating — is governed by contract; the protocol neither observes nor adjudicates it. Membership classes: source members (own content) and enrichment members (derived corpora, e.g. GDELT/CC-NEWS bots — metadata-only exposure, transformation logic open and auditable).
|
||||
|
||||
7. Reference Implementation — frxd
|
||||
|
||||
@@ -98,7 +98,7 @@ A conforming implementation: signs all messages with a key authorized for its id
|
||||
10. Open Issues
|
||||
|
||||
Consumer admission tier — automated/invite admission for distributed binaries without weakening the Sybil defense (MA policy, gates §7 adoption).
|
||||
Relay discovery and default-relay governance (shipped defaults are soft centralization; mitigate with multiple defaults + one-command self-host).
|
||||
Default-relay governance — registry-listed relays settle discovery, but who operates the defaults remains a soft centralization point (mitigate with multiple operators + one-command self-host).
|
||||
Delegation grant mechanism.
|
||||
Member-hosted identifiers — keys published in the member's own DNS instead of the MA registry; MA-hosted is normative until specified.
|
||||
MA anchor rollover — successor commitment and overlap for the registry signing key.
|
||||
|
||||
@@ -277,6 +277,7 @@ pub fn registry_init(dir: &Path) -> Result<()> {
|
||||
issued_at: now_ts(),
|
||||
ma_key: String::new(),
|
||||
members: Vec::new(),
|
||||
relays: Vec::new(),
|
||||
};
|
||||
let signed = registry::sign_registry(doc, &ma);
|
||||
registry::save_registry(®istry_path, &signed)?;
|
||||
@@ -390,11 +391,23 @@ pub fn registry_list(dir: &Path) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn registry_set_relays(dir: &Path, relays: &[String]) -> Result<()> {
|
||||
mutate_registry(dir, |doc| {
|
||||
doc.relays = relays.to_vec();
|
||||
Ok(())
|
||||
})?;
|
||||
println!("relays: {}", relays.join(" "));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn registry_show(dir: &Path) -> Result<()> {
|
||||
let (_, signed) = open_registry(dir)?;
|
||||
println!("ma_key {}", signed.doc.ma_key);
|
||||
println!("version {}", signed.doc.version);
|
||||
println!("issued_at {}", signed.doc.issued_at);
|
||||
for relay in &signed.doc.relays {
|
||||
println!("relay {relay}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
+33
-2
@@ -70,6 +70,16 @@ enum Command {
|
||||
listen: String,
|
||||
#[arg(long, default_value_t = relay::DEFAULT_CAPACITY)]
|
||||
capacity: usize,
|
||||
#[arg(long = "peer")]
|
||||
peers: Vec<String>,
|
||||
#[arg(long)]
|
||||
url: Option<String>,
|
||||
#[arg(long)]
|
||||
registry: Option<String>,
|
||||
#[arg(long)]
|
||||
ma_key: Option<String>,
|
||||
#[arg(long, default_value_t = 3)]
|
||||
max_hops: usize,
|
||||
},
|
||||
Status,
|
||||
Member {
|
||||
@@ -147,6 +157,10 @@ enum RegistryCommand {
|
||||
id: String,
|
||||
},
|
||||
List,
|
||||
SetRelays {
|
||||
#[arg(required = true)]
|
||||
relays: Vec<String>,
|
||||
},
|
||||
Show,
|
||||
Serve {
|
||||
#[arg(long, default_value = "127.0.0.1:7800")]
|
||||
@@ -231,10 +245,26 @@ async fn main() -> Result<()> {
|
||||
println!("control API POST http://{}/v1/local/query", handle.addr);
|
||||
tokio::signal::ctrl_c().await?;
|
||||
}
|
||||
Command::Relay { listen, capacity } => {
|
||||
Command::Relay {
|
||||
listen,
|
||||
capacity,
|
||||
peers,
|
||||
url,
|
||||
registry,
|
||||
ma_key,
|
||||
max_hops,
|
||||
} => {
|
||||
let options = relay::RelayOptions {
|
||||
capacity,
|
||||
peers,
|
||||
url,
|
||||
registry,
|
||||
ma_key,
|
||||
max_hops,
|
||||
};
|
||||
let (listener, addr) = relay::bind(&listen).await?;
|
||||
println!("relay listening on http://{addr} (capacity {capacity})");
|
||||
relay::run(listener, capacity).await?;
|
||||
relay::run(listener, options).await?;
|
||||
}
|
||||
Command::Status => commands::status(&cli.config).await?,
|
||||
Command::Member { command } => match command {
|
||||
@@ -271,6 +301,7 @@ async fn main() -> Result<()> {
|
||||
}
|
||||
RegistryCommand::Remove { id } => commands::registry_remove(&dir, &id)?,
|
||||
RegistryCommand::List => commands::registry_list(&dir)?,
|
||||
RegistryCommand::SetRelays { relays } => commands::registry_set_relays(&dir, &relays)?,
|
||||
RegistryCommand::Show => commands::registry_show(&dir)?,
|
||||
RegistryCommand::Serve { listen } => commands::registry_serve(&dir, &listen).await?,
|
||||
},
|
||||
|
||||
+41
-115
@@ -23,7 +23,7 @@ use crate::message::{
|
||||
AggregateBody, Envelope, QueryBody, ResponseBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY,
|
||||
TYPE_RESPONSE, build_response, timestamp_is_fresh,
|
||||
};
|
||||
use crate::registry::{SignedRegistry, authorized_keys, load_registry, verify_registry};
|
||||
use crate::registry::Watcher;
|
||||
|
||||
#[derive(Default)]
|
||||
struct Aggregates {
|
||||
@@ -37,8 +37,7 @@ pub struct Node {
|
||||
index: LocalIndex,
|
||||
members: RwLock<Vec<Member>>,
|
||||
members_mtime: Mutex<Option<SystemTime>>,
|
||||
registry: Mutex<Option<SignedRegistry>>,
|
||||
registry_mtime: Mutex<Option<SystemTime>>,
|
||||
registry: Option<Arc<Watcher>>,
|
||||
aggregates: Mutex<Aggregates>,
|
||||
seen: Mutex<HashSet<String>>,
|
||||
pending: Mutex<HashMap<String, Vec<(String, ResponseBody)>>>,
|
||||
@@ -122,14 +121,28 @@ impl Node {
|
||||
.timeout(Duration::from_secs(15))
|
||||
.build()
|
||||
.context("building http client")?;
|
||||
let node = Arc::new(Self {
|
||||
let registry = match (
|
||||
config.node.registry.as_deref(),
|
||||
config.node.ma_key.as_deref(),
|
||||
) {
|
||||
(Some(source), Some(ma_key)) => Some(Watcher::new(
|
||||
source,
|
||||
ma_key,
|
||||
Some(config.registry_cache_path()),
|
||||
)),
|
||||
(Some(_), None) => return Err(anyhow!("registry configured without ma_key")),
|
||||
_ => None,
|
||||
};
|
||||
if let Some(watcher) = ®istry {
|
||||
watcher.load_initial();
|
||||
}
|
||||
Ok(Arc::new(Self {
|
||||
config,
|
||||
key,
|
||||
index,
|
||||
members: RwLock::new(members),
|
||||
members_mtime: Mutex::new(members_mtime),
|
||||
registry: Mutex::new(None),
|
||||
registry_mtime: Mutex::new(None),
|
||||
registry,
|
||||
aggregates: Mutex::new(Aggregates::default()),
|
||||
seen: Mutex::new(HashSet::new()),
|
||||
pending: Mutex::new(HashMap::new()),
|
||||
@@ -137,90 +150,7 @@ impl Node {
|
||||
client,
|
||||
sent: AtomicU64::new(0),
|
||||
received: AtomicU64::new(0),
|
||||
});
|
||||
node.load_initial_registry();
|
||||
Ok(node)
|
||||
}
|
||||
|
||||
fn load_initial_registry(&self) {
|
||||
let Some(source) = &self.config.node.registry else {
|
||||
return;
|
||||
};
|
||||
let result = if source.starts_with("http://") || source.starts_with("https://") {
|
||||
load_registry(&self.config.registry_cache_path())
|
||||
} else {
|
||||
let path = std::path::PathBuf::from(source);
|
||||
let loaded = load_registry(&path);
|
||||
if loaded.is_ok() {
|
||||
let mtime = fs::metadata(&path)
|
||||
.and_then(|metadata| metadata.modified())
|
||||
.ok();
|
||||
*self.registry_mtime.lock().expect("registry mtime lock") = mtime;
|
||||
}
|
||||
loaded
|
||||
};
|
||||
if let Ok(signed) = result {
|
||||
if let Err(error) = self.verify_and_apply_registry(signed) {
|
||||
eprintln!("cached registry rejected: {error}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn verify_and_apply_registry(&self, signed: SignedRegistry) -> Result<()> {
|
||||
let ma_key = self
|
||||
.config
|
||||
.node
|
||||
.ma_key
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow!("registry configured without ma_key"))?;
|
||||
verify_registry(&signed, ma_key)?;
|
||||
{
|
||||
let current = self.registry.lock().expect("registry lock");
|
||||
if let Some(existing) = current.as_ref() {
|
||||
if signed.doc.version <= existing.doc.version {
|
||||
return Err(anyhow!("registry version rollback rejected"));
|
||||
}
|
||||
}
|
||||
}
|
||||
*self.registry.lock().expect("registry lock") = Some(signed);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn refresh_registry(&self) {
|
||||
let Some(source) = &self.config.node.registry else {
|
||||
return;
|
||||
};
|
||||
if source.starts_with("http://") || source.starts_with("https://") {
|
||||
return;
|
||||
}
|
||||
let path = std::path::PathBuf::from(source);
|
||||
let mtime = fs::metadata(&path)
|
||||
.and_then(|metadata| metadata.modified())
|
||||
.ok();
|
||||
{
|
||||
let last = self.registry_mtime.lock().expect("registry mtime lock");
|
||||
if *last == mtime {
|
||||
return;
|
||||
}
|
||||
}
|
||||
let Ok(signed) = load_registry(&path) else {
|
||||
return;
|
||||
};
|
||||
*self.registry_mtime.lock().expect("registry mtime lock") = mtime;
|
||||
if let Err(error) = self.verify_and_apply_registry(signed) {
|
||||
eprintln!("registry update rejected: {error}");
|
||||
}
|
||||
}
|
||||
|
||||
async fn fetch_registry(&self) -> Result<()> {
|
||||
let Some(url) = &self.config.node.registry else {
|
||||
return Ok(());
|
||||
};
|
||||
let response = self.client.get(url).send().await?;
|
||||
let signed: SignedRegistry = response.json().await?;
|
||||
self.verify_and_apply_registry(signed.clone())?;
|
||||
crate::registry::save_registry(&self.config.registry_cache_path(), &signed)?;
|
||||
Ok(())
|
||||
}))
|
||||
}
|
||||
|
||||
fn refresh_members(&self) {
|
||||
@@ -311,23 +241,31 @@ impl Node {
|
||||
.with_context(|| format!("binding {}", node.config.node.listen))?;
|
||||
let addr = listener.local_addr()?;
|
||||
let mut tasks = Vec::new();
|
||||
if let Some(source) = &node.config.node.registry {
|
||||
if source.starts_with("http://") || source.starts_with("https://") {
|
||||
if let Err(error) = node.fetch_registry().await {
|
||||
if let Some(watcher) = node.registry.clone() {
|
||||
if watcher.is_url() {
|
||||
if let Err(error) = watcher.fetch(&node.client).await {
|
||||
eprintln!("initial registry fetch failed: {error}");
|
||||
}
|
||||
let node_for_registry = node.clone();
|
||||
let client = node.client.clone();
|
||||
tasks.push(tokio::spawn(async move {
|
||||
loop {
|
||||
if let Err(error) = node_for_registry.fetch_registry().await {
|
||||
tokio::time::sleep(Duration::from_secs(60)).await;
|
||||
if let Err(error) = watcher.fetch(&client).await {
|
||||
eprintln!("registry fetch failed: {error}");
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(60)).await;
|
||||
}
|
||||
}));
|
||||
}
|
||||
}
|
||||
for relay in node.config.node.relays.clone() {
|
||||
let relays = if node.config.node.relays.is_empty() {
|
||||
node.registry
|
||||
.as_ref()
|
||||
.map(|watcher| watcher.relays())
|
||||
.unwrap_or_default()
|
||||
} else {
|
||||
node.config.node.relays.clone()
|
||||
};
|
||||
for relay in relays {
|
||||
tasks.push(tokio::spawn(poll_relay(node.clone(), relay)));
|
||||
}
|
||||
let app = router(node.clone());
|
||||
@@ -456,18 +394,11 @@ impl Node {
|
||||
return;
|
||||
}
|
||||
self.refresh_members();
|
||||
self.refresh_registry();
|
||||
let (class, listed) = if self.config.node.registry.is_some() {
|
||||
let registry = self.registry.lock().expect("registry lock");
|
||||
let authorized = registry
|
||||
.as_ref()
|
||||
.map(|signed| authorized_keys(signed, now_ts()));
|
||||
match authorized {
|
||||
Some(map) => match map.get(&envelope.key) {
|
||||
Some((id, class)) if id == &envelope.from => (Some(class.clone()), true),
|
||||
_ => (None, false),
|
||||
},
|
||||
None => (None, false),
|
||||
let (class, listed) = if let Some(watcher) = &self.registry {
|
||||
watcher.refresh_if_changed();
|
||||
match watcher.authorized(&envelope.key, now_ts()) {
|
||||
Some((id, class)) if id == envelope.from => (Some(class), true),
|
||||
_ => (None, false),
|
||||
}
|
||||
} else {
|
||||
let members = self.members.read().expect("members lock");
|
||||
@@ -761,12 +692,7 @@ async fn local_query(
|
||||
|
||||
async fn local_status(State(node): State<Arc<Node>>) -> Response {
|
||||
let aggregates = node.aggregate_for(¤t_period(), None);
|
||||
let registry_version = node
|
||||
.registry
|
||||
.lock()
|
||||
.expect("registry lock")
|
||||
.as_ref()
|
||||
.map(|signed| signed.doc.version);
|
||||
let registry_version = node.registry.as_ref().and_then(|watcher| watcher.version());
|
||||
Json(json!({
|
||||
"name": node.config.node.name,
|
||||
"id": node.config.node.id,
|
||||
|
||||
+127
-1
@@ -1,6 +1,8 @@
|
||||
use std::collections::HashMap;
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::SystemTime;
|
||||
|
||||
use anyhow::{Context, Result, anyhow};
|
||||
use serde::{Deserialize, Serialize};
|
||||
@@ -35,6 +37,8 @@ pub struct RegistryDoc {
|
||||
pub ma_key: String,
|
||||
#[serde(default)]
|
||||
pub members: Vec<RegistryMember>,
|
||||
#[serde(default)]
|
||||
pub relays: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@@ -75,6 +79,127 @@ pub fn save_registry(path: &Path, signed: &SignedRegistry) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub struct Watcher {
|
||||
source: String,
|
||||
ma_key: String,
|
||||
cache_path: Option<PathBuf>,
|
||||
current: Mutex<Option<SignedRegistry>>,
|
||||
mtime: Mutex<Option<SystemTime>>,
|
||||
}
|
||||
|
||||
impl Watcher {
|
||||
pub fn new(source: &str, ma_key: &str, cache_path: Option<PathBuf>) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
source: source.to_string(),
|
||||
ma_key: ma_key.to_string(),
|
||||
cache_path,
|
||||
current: Mutex::new(None),
|
||||
mtime: Mutex::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn is_url(&self) -> bool {
|
||||
self.source.starts_with("http://") || self.source.starts_with("https://")
|
||||
}
|
||||
|
||||
pub fn load_initial(&self) {
|
||||
let loaded = if self.is_url() {
|
||||
match &self.cache_path {
|
||||
Some(cache) => load_registry(cache),
|
||||
None => Err(anyhow!("no cache path for URL registry")),
|
||||
}
|
||||
} else {
|
||||
let path = PathBuf::from(&self.source);
|
||||
let loaded = load_registry(&path);
|
||||
if loaded.is_ok() {
|
||||
*self.mtime.lock().expect("registry mtime lock") = fs::metadata(&path)
|
||||
.and_then(|metadata| metadata.modified())
|
||||
.ok();
|
||||
}
|
||||
loaded
|
||||
};
|
||||
if let Ok(signed) = loaded {
|
||||
if let Err(error) = self.apply(signed) {
|
||||
eprintln!("cached registry rejected: {error}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn refresh_if_changed(&self) {
|
||||
if self.is_url() {
|
||||
return;
|
||||
}
|
||||
let path = PathBuf::from(&self.source);
|
||||
let mtime = fs::metadata(&path)
|
||||
.and_then(|metadata| metadata.modified())
|
||||
.ok();
|
||||
{
|
||||
let last = self.mtime.lock().expect("registry mtime lock");
|
||||
if *last == mtime {
|
||||
return;
|
||||
}
|
||||
}
|
||||
let Ok(signed) = load_registry(&path) else {
|
||||
return;
|
||||
};
|
||||
*self.mtime.lock().expect("registry mtime lock") = mtime;
|
||||
if let Err(error) = self.apply(signed) {
|
||||
eprintln!("registry update rejected: {error}");
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch(&self, client: &reqwest::Client) -> Result<()> {
|
||||
if !self.is_url() {
|
||||
return Ok(());
|
||||
}
|
||||
let response = client.get(&self.source).send().await?;
|
||||
let signed: SignedRegistry = response.json().await?;
|
||||
self.apply(signed.clone())?;
|
||||
if let Some(cache) = &self.cache_path {
|
||||
save_registry(cache, &signed)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply(&self, signed: SignedRegistry) -> Result<()> {
|
||||
verify_registry(&signed, &self.ma_key)?;
|
||||
{
|
||||
let current = self.current.lock().expect("registry lock");
|
||||
if let Some(existing) = current.as_ref() {
|
||||
if signed.doc.version <= existing.doc.version {
|
||||
return Err(anyhow!("registry version rollback rejected"));
|
||||
}
|
||||
}
|
||||
}
|
||||
*self.current.lock().expect("registry lock") = Some(signed);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn authorized(&self, key: &str, now: u64) -> Option<(String, String)> {
|
||||
let current = self.current.lock().expect("registry lock");
|
||||
current
|
||||
.as_ref()
|
||||
.and_then(|signed| authorized_keys(signed, now).remove(key))
|
||||
}
|
||||
|
||||
pub fn version(&self) -> Option<u64> {
|
||||
self.current
|
||||
.lock()
|
||||
.expect("registry lock")
|
||||
.as_ref()
|
||||
.map(|signed| signed.doc.version)
|
||||
}
|
||||
|
||||
pub fn relays(&self) -> Vec<String> {
|
||||
self.current
|
||||
.lock()
|
||||
.expect("registry lock")
|
||||
.as_ref()
|
||||
.map(|signed| signed.doc.relays.clone())
|
||||
.unwrap_or_default()
|
||||
}
|
||||
}
|
||||
|
||||
pub fn authorized_keys(signed: &SignedRegistry, now: u64) -> HashMap<String, (String, String)> {
|
||||
let mut authorized = HashMap::new();
|
||||
for member in &signed.doc.members {
|
||||
@@ -117,6 +242,7 @@ mod tests {
|
||||
issued_at: now_ts(),
|
||||
ma_key: String::new(),
|
||||
members,
|
||||
relays: Vec::new(),
|
||||
},
|
||||
key,
|
||||
)
|
||||
|
||||
+229
-38
@@ -3,7 +3,7 @@ use std::net::SocketAddr;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use anyhow::Result;
|
||||
use anyhow::{Result, anyhow};
|
||||
use axum::extract::{Query, State};
|
||||
use axum::http::{StatusCode, header};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
@@ -15,13 +15,40 @@ use tokio::net::TcpListener;
|
||||
|
||||
use crate::crypto::{now_ts, poll_signing_bytes, random_nonce, verify_signature};
|
||||
use crate::message::{Envelope, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE, timestamp_is_fresh};
|
||||
use crate::registry::Watcher;
|
||||
|
||||
pub const DEFAULT_CAPACITY: usize = 256;
|
||||
const CHALLENGE_TTL: Duration = Duration::from_secs(120);
|
||||
const SEEN_TTL: Duration = Duration::from_secs(30);
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct RelayOptions {
|
||||
pub capacity: usize,
|
||||
pub peers: Vec<String>,
|
||||
pub url: Option<String>,
|
||||
pub registry: Option<String>,
|
||||
pub ma_key: Option<String>,
|
||||
pub max_hops: usize,
|
||||
}
|
||||
|
||||
impl Default for RelayOptions {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
capacity: DEFAULT_CAPACITY,
|
||||
peers: Vec::new(),
|
||||
url: None,
|
||||
registry: None,
|
||||
ma_key: None,
|
||||
max_hops: 3,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct MemberQueue {
|
||||
items: VecDeque<(u64, Envelope)>,
|
||||
lagging: bool,
|
||||
missed: u64,
|
||||
}
|
||||
|
||||
struct Challenge {
|
||||
@@ -32,57 +59,136 @@ struct Challenge {
|
||||
struct Inner {
|
||||
members: HashMap<String, MemberQueue>,
|
||||
challenges: HashMap<String, Challenge>,
|
||||
seen: HashMap<String, Instant>,
|
||||
seq: u64,
|
||||
}
|
||||
|
||||
pub struct Relay {
|
||||
inner: Mutex<Inner>,
|
||||
capacity: usize,
|
||||
options: RelayOptions,
|
||||
registry: Option<Arc<Watcher>>,
|
||||
client: reqwest::Client,
|
||||
}
|
||||
|
||||
impl Relay {
|
||||
pub fn new(capacity: usize) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
pub fn new(options: RelayOptions) -> Result<Arc<Self>> {
|
||||
let registry = match (options.registry.as_deref(), options.ma_key.as_deref()) {
|
||||
(Some(source), Some(ma_key)) => Some(Watcher::new(source, ma_key, None)),
|
||||
(Some(_), None) => return Err(anyhow!("registry configured without ma_key")),
|
||||
_ => None,
|
||||
};
|
||||
if !options.peers.is_empty() && options.url.is_none() {
|
||||
return Err(anyhow!("--url is required when --peer is set"));
|
||||
}
|
||||
let relay = Arc::new(Self {
|
||||
inner: Mutex::new(Inner {
|
||||
members: HashMap::new(),
|
||||
challenges: HashMap::new(),
|
||||
seen: HashMap::new(),
|
||||
seq: 0,
|
||||
}),
|
||||
capacity,
|
||||
})
|
||||
options,
|
||||
registry,
|
||||
client: reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(5))
|
||||
.build()?,
|
||||
});
|
||||
if let Some(watcher) = &relay.registry {
|
||||
watcher.load_initial();
|
||||
}
|
||||
Ok(relay)
|
||||
}
|
||||
|
||||
fn push(&self, targets: Option<&str>, envelope: Envelope) -> Result<usize, StatusCode> {
|
||||
let mut inner = self.inner.lock().expect("relay lock");
|
||||
if inner
|
||||
.members
|
||||
.values()
|
||||
.any(|q| q.items.len() >= self.capacity)
|
||||
{
|
||||
return Err(StatusCode::TOO_MANY_REQUESTS);
|
||||
fn admit(&self, envelope: &Envelope) -> bool {
|
||||
match &self.registry {
|
||||
Some(watcher) => {
|
||||
watcher.refresh_if_changed();
|
||||
watcher.authorized(&envelope.key, now_ts()).is_some()
|
||||
}
|
||||
None => true,
|
||||
}
|
||||
}
|
||||
|
||||
fn mark_seen(&self, signature: &str) -> bool {
|
||||
let mut inner = self.inner.lock().expect("relay lock");
|
||||
let now = Instant::now();
|
||||
inner
|
||||
.seen
|
||||
.retain(|_, at| now.duration_since(*at) < SEEN_TTL);
|
||||
if inner.seen.contains_key(signature) {
|
||||
return false;
|
||||
}
|
||||
inner.seen.insert(signature.to_string(), now);
|
||||
true
|
||||
}
|
||||
|
||||
fn push_local(&self, target: Option<&str>, envelope: Envelope) -> Result<usize, StatusCode> {
|
||||
let mut inner = self.inner.lock().expect("relay lock");
|
||||
inner.seq += 1;
|
||||
let seq = inner.seq;
|
||||
match targets {
|
||||
match target {
|
||||
Some(member) => {
|
||||
let Some(queue) = inner.members.get_mut(member) else {
|
||||
return Err(StatusCode::NOT_FOUND);
|
||||
};
|
||||
if queue.items.len() >= self.options.capacity {
|
||||
queue.lagging = true;
|
||||
queue.missed += 1;
|
||||
return Err(StatusCode::TOO_MANY_REQUESTS);
|
||||
}
|
||||
queue.items.push_back((seq, envelope));
|
||||
Ok(1)
|
||||
}
|
||||
None => {
|
||||
let from = envelope.key.clone();
|
||||
inner.members.entry(from).or_default();
|
||||
let publisher = envelope.key.clone();
|
||||
inner.members.entry(publisher).or_default();
|
||||
let mut delivered = 0;
|
||||
for queue in inner.members.values_mut() {
|
||||
queue.items.push_back((seq, envelope.clone()));
|
||||
delivered += 1;
|
||||
if queue.items.len() >= self.options.capacity {
|
||||
queue.lagging = true;
|
||||
queue.missed += 1;
|
||||
} else {
|
||||
queue.items.push_back((seq, envelope.clone()));
|
||||
delivered += 1;
|
||||
}
|
||||
}
|
||||
Ok(delivered)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn forward(&self, envelope: Envelope, origin: Option<&str>, hops: usize) {
|
||||
if self.options.peers.is_empty() {
|
||||
return;
|
||||
}
|
||||
let Some(url) = self.options.url.clone() else {
|
||||
return;
|
||||
};
|
||||
let peers: Vec<String> = self
|
||||
.options
|
||||
.peers
|
||||
.iter()
|
||||
.filter(|peer| Some(peer.as_str()) != origin)
|
||||
.cloned()
|
||||
.collect();
|
||||
if peers.is_empty() {
|
||||
return;
|
||||
}
|
||||
let client = self.client.clone();
|
||||
tokio::spawn(async move {
|
||||
for peer in peers {
|
||||
let target = format!(
|
||||
"{}/v1/federation?origin={}&hops={}",
|
||||
peer.trim_end_matches('/'),
|
||||
url,
|
||||
hops
|
||||
);
|
||||
if let Err(error) = client.post(&target).json(&envelope).send().await {
|
||||
eprintln!("federation forward to {peer} failed: {error}");
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
pub fn router(relay: Arc<Relay>) -> Router {
|
||||
@@ -90,13 +196,30 @@ pub fn router(relay: Arc<Relay>) -> Router {
|
||||
.route("/health", get(health))
|
||||
.route("/v1/challenge", get(challenge))
|
||||
.route("/v1/publish", post(publish))
|
||||
.route("/v1/federation", post(federation))
|
||||
.route("/v1/unicast", post(unicast))
|
||||
.route("/v1/poll", get(poll))
|
||||
.with_state(relay)
|
||||
}
|
||||
|
||||
pub async fn run(listener: TcpListener, capacity: usize) -> Result<()> {
|
||||
let relay = Relay::new(capacity);
|
||||
pub async fn run(listener: TcpListener, options: RelayOptions) -> Result<()> {
|
||||
let relay = Relay::new(options)?;
|
||||
if let Some(watcher) = relay.registry.clone() {
|
||||
if watcher.is_url() {
|
||||
let client = relay.client.clone();
|
||||
if let Err(error) = watcher.fetch(&client).await {
|
||||
eprintln!("initial registry fetch failed: {error}");
|
||||
}
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
tokio::time::sleep(Duration::from_secs(60)).await;
|
||||
if let Err(error) = watcher.fetch(&client).await {
|
||||
eprintln!("registry fetch failed: {error}");
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
axum::serve(listener, router(relay)).await?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -152,14 +275,65 @@ async fn publish(State(relay): State<Arc<Relay>>, Json(envelope): Json<Envelope>
|
||||
if envelope.msg_type != TYPE_QUERY {
|
||||
return bad_request("relay carries broadcast queries only");
|
||||
}
|
||||
match relay.push(None, envelope) {
|
||||
Ok(delivered) => (
|
||||
StatusCode::ACCEPTED,
|
||||
Json(json!({ "delivered": delivered })),
|
||||
)
|
||||
.into_response(),
|
||||
Err(status) => backpressure(status),
|
||||
if !relay.admit(&envelope) {
|
||||
return bad_request("sender not admitted");
|
||||
}
|
||||
relay.mark_seen(&envelope.sig);
|
||||
let delivered = relay.push_local(None, envelope.clone()).unwrap_or(0);
|
||||
relay.forward(envelope, None, 1);
|
||||
(
|
||||
StatusCode::ACCEPTED,
|
||||
Json(json!({ "delivered": delivered })),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct FederationParams {
|
||||
origin: Option<String>,
|
||||
hops: Option<usize>,
|
||||
}
|
||||
|
||||
async fn federation(
|
||||
State(relay): State<Arc<Relay>>,
|
||||
Query(params): Query<FederationParams>,
|
||||
Json(envelope): Json<Envelope>,
|
||||
) -> Response {
|
||||
let Some(origin) = params.origin.clone() else {
|
||||
return bad_request("origin required");
|
||||
};
|
||||
if !relay.options.peers.iter().any(|peer| peer == &origin) {
|
||||
return bad_request("unknown peer origin");
|
||||
}
|
||||
if envelope.verify().is_err() {
|
||||
return bad_request("invalid signature");
|
||||
}
|
||||
if !timestamp_is_fresh(envelope.ts, now_ts()) {
|
||||
return bad_request("stale timestamp");
|
||||
}
|
||||
if envelope.msg_type != TYPE_QUERY {
|
||||
return bad_request("relay carries broadcast queries only");
|
||||
}
|
||||
if !relay.admit(&envelope) {
|
||||
return bad_request("sender not admitted");
|
||||
}
|
||||
if !relay.mark_seen(&envelope.sig) {
|
||||
return (
|
||||
StatusCode::ACCEPTED,
|
||||
Json(json!({ "delivered": 0, "duplicate": true })),
|
||||
)
|
||||
.into_response();
|
||||
}
|
||||
let delivered = relay.push_local(None, envelope.clone()).unwrap_or(0);
|
||||
let hops = params.hops.unwrap_or(1);
|
||||
if hops < relay.options.max_hops {
|
||||
relay.forward(envelope, Some(&origin), hops + 1);
|
||||
}
|
||||
(
|
||||
StatusCode::ACCEPTED,
|
||||
Json(json!({ "delivered": delivered })),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
@@ -181,14 +355,17 @@ async fn unicast(
|
||||
if envelope.msg_type != TYPE_RESPONSE && envelope.msg_type != TYPE_AGGREGATE {
|
||||
return bad_request("unicast carries responses and aggregates only");
|
||||
}
|
||||
match relay.push(Some(¶ms.to), envelope) {
|
||||
if !relay.admit(&envelope) {
|
||||
return bad_request("sender not admitted");
|
||||
}
|
||||
match relay.push_local(Some(¶ms.to), envelope) {
|
||||
Ok(_) => (StatusCode::OK, Json(json!({ "delivered": true }))).into_response(),
|
||||
Err(StatusCode::NOT_FOUND) => (
|
||||
StatusCode::NOT_FOUND,
|
||||
Json(json!({ "error": "member not connected" })),
|
||||
)
|
||||
.into_response(),
|
||||
Err(status) => backpressure(status),
|
||||
Err(status) => backpressure(status, 0),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -225,15 +402,29 @@ async fn poll(State(relay): State<Arc<Relay>>, Query(params): Query<PollParams>)
|
||||
let timeout = Duration::from_millis(params.timeout_ms.unwrap_or(25_000).min(60_000));
|
||||
let deadline = Instant::now() + timeout;
|
||||
loop {
|
||||
let batch: Vec<Envelope> = {
|
||||
let (batch, lagged): (Vec<Envelope>, Option<u64>) = {
|
||||
let mut inner = relay.inner.lock().expect("relay lock");
|
||||
let queue = inner.members.entry(params.member.clone()).or_default();
|
||||
queue
|
||||
.items
|
||||
.drain(..)
|
||||
.map(|(_, envelope)| envelope)
|
||||
.collect()
|
||||
if queue.lagging {
|
||||
let missed = queue.missed;
|
||||
queue.lagging = false;
|
||||
queue.missed = 0;
|
||||
queue.items.clear();
|
||||
(Vec::new(), Some(missed))
|
||||
} else {
|
||||
(
|
||||
queue
|
||||
.items
|
||||
.drain(..)
|
||||
.map(|(_, envelope)| envelope)
|
||||
.collect(),
|
||||
None,
|
||||
)
|
||||
}
|
||||
};
|
||||
if let Some(missed) = lagged {
|
||||
return backpressure(StatusCode::TOO_MANY_REQUESTS, missed);
|
||||
}
|
||||
if !batch.is_empty() {
|
||||
return (StatusCode::OK, Json(json!({ "messages": batch }))).into_response();
|
||||
}
|
||||
@@ -252,11 +443,11 @@ fn unauthorized(message: &str) -> Response {
|
||||
(StatusCode::UNAUTHORIZED, Json(json!({ "error": message }))).into_response()
|
||||
}
|
||||
|
||||
fn backpressure(status: StatusCode) -> Response {
|
||||
fn backpressure(status: StatusCode, missed: u64) -> Response {
|
||||
(
|
||||
status,
|
||||
[(header::RETRY_AFTER, "1")],
|
||||
Json(json!({ "error": "transport backpressure" })),
|
||||
Json(json!({ "error": "lagging", "missed": missed })),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
+5
-1
@@ -49,7 +49,11 @@ pub async fn spawn_relay() -> String {
|
||||
pub async fn spawn_relay_with_capacity(capacity: usize) -> String {
|
||||
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
|
||||
tokio::spawn(async move {
|
||||
let _ = relay::run(listener, capacity).await;
|
||||
let options = relay::RelayOptions {
|
||||
capacity,
|
||||
..Default::default()
|
||||
};
|
||||
let _ = relay::run(listener, options).await;
|
||||
});
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
+52
-13
@@ -478,38 +478,53 @@ async fn stale_envelopes_are_rejected() {
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn backpressure_is_visible_and_recoverable() {
|
||||
async fn lagging_member_is_isolated_and_reported() {
|
||||
let relay_url = spawn_relay_with_capacity(1).await;
|
||||
let http = client();
|
||||
let member = Keypair::generate();
|
||||
let active = Keypair::generate();
|
||||
let slow = Keypair::generate();
|
||||
let publisher = Keypair::generate();
|
||||
register(&http, &relay_url, &member).await;
|
||||
register(&http, &relay_url, &active).await;
|
||||
register(&http, &relay_url, &slow).await;
|
||||
|
||||
let first = query_envelope(&publisher, &publisher.public_hex(), "one", 5);
|
||||
assert_eq!(
|
||||
publish(&http, &relay_url, &first).await.status(),
|
||||
reqwest::StatusCode::ACCEPTED
|
||||
);
|
||||
assert_eq!(
|
||||
poll_messages(&http, &relay_url, &active, 300).await.len(),
|
||||
1
|
||||
);
|
||||
|
||||
let second = query_envelope(&publisher, &publisher.public_hex(), "two", 5);
|
||||
let response = publish(&http, &relay_url, &second).await;
|
||||
assert_eq!(
|
||||
publish(&http, &relay_url, &second).await.status(),
|
||||
reqwest::StatusCode::ACCEPTED,
|
||||
"a lagging subscriber must not stall publishers"
|
||||
);
|
||||
assert_eq!(
|
||||
poll_messages(&http, &relay_url, &active, 300).await.len(),
|
||||
1
|
||||
);
|
||||
|
||||
let response = poll(&http, &relay_url, &slow, 300).await;
|
||||
assert_eq!(response.status(), reqwest::StatusCode::TOO_MANY_REQUESTS);
|
||||
assert_eq!(
|
||||
response
|
||||
.headers()
|
||||
.get("retry-after")
|
||||
.and_then(|v| v.to_str().ok()),
|
||||
.and_then(|value| value.to_str().ok()),
|
||||
Some("1")
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
poll_messages(&http, &relay_url, &member, 300).await.len(),
|
||||
1
|
||||
let payload: Value = response.json().await.unwrap();
|
||||
assert!(
|
||||
payload.get("missed").and_then(Value::as_u64).unwrap_or(0) >= 1,
|
||||
"lagging member must be told how much it missed: {payload}"
|
||||
);
|
||||
assert_eq!(
|
||||
poll_messages(&http, &relay_url, &publisher, 300)
|
||||
.await
|
||||
.len(),
|
||||
1
|
||||
poll(&http, &relay_url, &slow, 50).await.status(),
|
||||
reqwest::StatusCode::NO_CONTENT
|
||||
);
|
||||
|
||||
let third = query_envelope(&publisher, &publisher.public_hex(), "three", 5);
|
||||
@@ -517,6 +532,30 @@ async fn backpressure_is_visible_and_recoverable() {
|
||||
publish(&http, &relay_url, &third).await.status(),
|
||||
reqwest::StatusCode::ACCEPTED
|
||||
);
|
||||
assert_eq!(poll_messages(&http, &relay_url, &slow, 300).await.len(), 1);
|
||||
|
||||
let fourth = query_envelope(&publisher, &publisher.public_hex(), "four", 5);
|
||||
assert_eq!(
|
||||
publish(&http, &relay_url, &fourth).await.status(),
|
||||
reqwest::StatusCode::ACCEPTED
|
||||
);
|
||||
let fifth = query_envelope(&publisher, &publisher.public_hex(), "five", 5);
|
||||
assert_eq!(
|
||||
publish(&http, &relay_url, &fifth).await.status(),
|
||||
reqwest::StatusCode::ACCEPTED
|
||||
);
|
||||
let direct = test_envelope(
|
||||
&publisher,
|
||||
TYPE_RESPONSE,
|
||||
json!({"qid": "q", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
|
||||
);
|
||||
assert_eq!(
|
||||
unicast(&http, &relay_url, &slow.public_hex(), &direct)
|
||||
.await
|
||||
.status(),
|
||||
reqwest::StatusCode::TOO_MANY_REQUESTS,
|
||||
"unicast to a lagging member must be visibly rejected"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
mod common;
|
||||
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
use std::time::Duration;
|
||||
|
||||
use common::{
|
||||
client, collection, config_for, messages_of_type, poll, poll_messages, publish, query_envelope,
|
||||
register, test_envelope,
|
||||
};
|
||||
use frxd::crypto::{Keypair, now_ts};
|
||||
use frxd::index::LocalIndex;
|
||||
use frxd::message::{EXPOSURE_FULL, TYPE_QUERY, TYPE_RESPONSE};
|
||||
use frxd::node::Node;
|
||||
use frxd::registry::{self, KeyEntry, RegistryDoc, RegistryMember, SignedRegistry};
|
||||
use frxd::relay::{self, RelayOptions};
|
||||
|
||||
async fn spawn_relay_with(options: RelayOptions) -> String {
|
||||
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
|
||||
tokio::spawn(async move {
|
||||
let _ = relay::run(listener, options).await;
|
||||
});
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
async fn federation_mesh() -> [String; 3] {
|
||||
let mut listeners = Vec::new();
|
||||
let mut urls = Vec::new();
|
||||
for _ in 0..3 {
|
||||
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
|
||||
listeners.push(listener);
|
||||
urls.push(format!("http://{addr}"));
|
||||
}
|
||||
let peers: Vec<Vec<String>> = vec![
|
||||
vec![urls[1].clone(), urls[2].clone()],
|
||||
vec![urls[0].clone(), urls[2].clone()],
|
||||
vec![urls[0].clone(), urls[1].clone()],
|
||||
];
|
||||
for (index, listener) in listeners.into_iter().enumerate() {
|
||||
let options = RelayOptions {
|
||||
url: Some(urls[index].clone()),
|
||||
peers: peers[index].clone(),
|
||||
..Default::default()
|
||||
};
|
||||
tokio::spawn(async move {
|
||||
let _ = relay::run(listener, options).await;
|
||||
});
|
||||
}
|
||||
[urls[0].clone(), urls[1].clone(), urls[2].clone()]
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn mesh_floods_queries_exactly_once() {
|
||||
let [r1, _r2, r3] = federation_mesh().await;
|
||||
let http = client();
|
||||
let member = Keypair::generate();
|
||||
let publisher = Keypair::generate();
|
||||
register(&http, &r3, &member).await;
|
||||
|
||||
let envelope = query_envelope(&publisher, &publisher.public_hex(), "mesh", 5);
|
||||
assert!(publish(&http, &r1, &envelope).await.status().is_success());
|
||||
|
||||
let messages = poll_messages(&http, &r3, &member, 900).await;
|
||||
assert_eq!(messages.len(), 1, "expected one flooded copy: {messages:?}");
|
||||
|
||||
tokio::time::sleep(Duration::from_millis(300)).await;
|
||||
let duplicates = poll_messages(&http, &r3, &member, 200).await;
|
||||
assert!(
|
||||
duplicates.is_empty(),
|
||||
"mesh delivered duplicates: {duplicates:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn federation_rejects_unknown_origin() {
|
||||
let [r1, _r2, _r3] = federation_mesh().await;
|
||||
let http = client();
|
||||
let key = Keypair::generate();
|
||||
let envelope = query_envelope(&key, &key.public_hex(), "spoof", 5);
|
||||
let response = http
|
||||
.post(format!(
|
||||
"{r1}/v1/federation?origin=http://127.0.0.1:1&hops=1"
|
||||
))
|
||||
.json(&envelope)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||
}
|
||||
|
||||
fn registry_doc(
|
||||
ma: &Keypair,
|
||||
members: Vec<RegistryMember>,
|
||||
relays: Vec<String>,
|
||||
version: u64,
|
||||
) -> SignedRegistry {
|
||||
registry::sign_registry(
|
||||
RegistryDoc {
|
||||
version,
|
||||
issued_at: now_ts(),
|
||||
ma_key: String::new(),
|
||||
members,
|
||||
relays,
|
||||
},
|
||||
ma,
|
||||
)
|
||||
}
|
||||
|
||||
fn member(id: &str, key: &Keypair, class: &str) -> RegistryMember {
|
||||
RegistryMember {
|
||||
id: id.to_string(),
|
||||
class: class.to_string(),
|
||||
keys: vec![KeyEntry {
|
||||
key: key.public_hex(),
|
||||
not_before: 0,
|
||||
not_after: None,
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn relay_admission_gates_unlisted_keys() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let ma = Keypair::generate();
|
||||
let alice = Keypair::generate();
|
||||
let registry_path = root.path().join("registry.json");
|
||||
registry::save_registry(
|
||||
®istry_path,
|
||||
®istry_doc(
|
||||
&ma,
|
||||
vec![member("alice.frx.example", &alice, "source")],
|
||||
vec![],
|
||||
1,
|
||||
),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let relay_url = spawn_relay_with(RelayOptions {
|
||||
registry: Some(registry_path.display().to_string()),
|
||||
ma_key: Some(ma.public_hex()),
|
||||
..Default::default()
|
||||
})
|
||||
.await;
|
||||
let http = client();
|
||||
|
||||
let accepted = query_envelope(&alice, "alice.frx.example", "rust", 5);
|
||||
assert_eq!(
|
||||
publish(&http, &relay_url, &accepted).await.status(),
|
||||
reqwest::StatusCode::ACCEPTED
|
||||
);
|
||||
|
||||
let stranger = Keypair::generate();
|
||||
let denied = query_envelope(&stranger, &stranger.public_hex(), "rust", 5);
|
||||
assert_eq!(
|
||||
publish(&http, &relay_url, &denied).await.status(),
|
||||
reqwest::StatusCode::BAD_REQUEST,
|
||||
"unlisted key must not publish"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn node_discovers_relays_from_registry() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let ma = Keypair::generate();
|
||||
let alice = Keypair::generate();
|
||||
|
||||
let (listener, relay_addr) = relay::bind("127.0.0.1:0").await.unwrap();
|
||||
tokio::spawn(async move {
|
||||
let _ = relay::run(listener, RelayOptions::default()).await;
|
||||
});
|
||||
let relay_url = format!("http://{relay_addr}");
|
||||
|
||||
let docs = root.path().join("bob-docs");
|
||||
fs::create_dir_all(&docs).unwrap();
|
||||
fs::write(docs.join("doc.txt"), "registry discovery rust document").unwrap();
|
||||
|
||||
let mut bob_config = config_for(&root.path().join("bob"), "bob", "http://127.0.0.1:1");
|
||||
bob_config.node.relays = Vec::new();
|
||||
bob_config.node.id = Some("bob.frx.example".to_string());
|
||||
let registry_path = root.path().join("registry.json");
|
||||
bob_config.node.registry = Some(registry_path.display().to_string());
|
||||
bob_config.node.ma_key = Some(ma.public_hex());
|
||||
bob_config.node.dev_bootstrap = false;
|
||||
registry::save_registry(
|
||||
®istry_path,
|
||||
®istry_doc(
|
||||
&ma,
|
||||
vec![
|
||||
member("alice.frx.example", &alice, "source"),
|
||||
member("bob.frx.example", &bob_config.load_key().unwrap(), "source"),
|
||||
],
|
||||
vec![relay_url.clone()],
|
||||
1,
|
||||
),
|
||||
)
|
||||
.unwrap();
|
||||
{
|
||||
let index = LocalIndex::open(&bob_config.index_dir()).unwrap();
|
||||
index
|
||||
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
|
||||
.unwrap();
|
||||
}
|
||||
let _bob = Node::start(bob_config).await.unwrap();
|
||||
|
||||
let http = client();
|
||||
register(&http, &relay_url, &alice).await;
|
||||
let envelope = query_envelope(&alice, "alice.frx.example", "rust", 5);
|
||||
assert!(
|
||||
publish(&http, &relay_url, &envelope)
|
||||
.await
|
||||
.status()
|
||||
.is_success()
|
||||
);
|
||||
|
||||
let mut responses = Vec::new();
|
||||
for _ in 0..5 {
|
||||
let messages = poll_messages(&http, &relay_url, &alice, 200).await;
|
||||
responses.extend(messages_of_type(&messages, TYPE_RESPONSE));
|
||||
if !responses.is_empty() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert_eq!(
|
||||
responses.len(),
|
||||
1,
|
||||
"node did not discover its relay from the registry"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn unknown_types_never_flood() {
|
||||
let [r1, _r2, _r3] = federation_mesh().await;
|
||||
let http = client();
|
||||
let key = Keypair::generate();
|
||||
let response = test_envelope(
|
||||
&key,
|
||||
TYPE_RESPONSE,
|
||||
serde_json::json!({"qid": "q", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
|
||||
);
|
||||
let result = http
|
||||
.post(format!(
|
||||
"{r1}/v1/federation?origin={}&hops=1",
|
||||
"http://unknown"
|
||||
))
|
||||
.json(&response)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(result.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||
let _ = TYPE_QUERY;
|
||||
}
|
||||
@@ -41,6 +41,7 @@ fn ma_registry(ma: &Keypair, members: Vec<RegistryMember>, version: u64) -> Sign
|
||||
issued_at: now_ts(),
|
||||
ma_key: String::new(),
|
||||
members,
|
||||
relays: Vec::new(),
|
||||
},
|
||||
ma,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user