Add Phase 1 frxd implementation with conformance test suite

This commit is contained in:
George Coles
2026-09-15 03:07:17 -04:00
parent 81ef27179b
commit 82a95fb907
23 changed files with 7295 additions and 3 deletions
+1
View File
@@ -0,0 +1 @@
/target
+21 -3
View File
@@ -1,8 +1,10 @@
# AGENTS.md # AGENTS.md
## Repo shape ## Repo shape
- Spec-only repo: `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.3) is the entire content. No code, README, manifests, tests, CI, or build tooling — don't look for or invent build/lint/test commands. - `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.3) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
- `frxd` (Rust: tokio, tantivy, axum, ed25519-dalek) is planned in §7 but does not exist here; treat it as design intent, not current structure. - `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
- Commands: `cargo build`, `cargo test` (62 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
- E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.
## Editing the spec ## Editing the spec
- Read all of `rfc.txt` before editing; it is the sole source of truth and is deliberately terse. - Read all of `rfc.txt` before editing; it is the sole source of truth and is deliberately terse.
@@ -12,10 +14,26 @@
- §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something. - §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something.
- Use the spec's vocabulary — member/querier/responder, citations/receipts/aggregates, source/enrichment members — not client/server or search-engine terms. - Use the spec's vocabulary — member/querier/responder, citations/receipts/aggregates, source/enrichment members — not client/server or search-engine terms.
## Implementation notes
- Envelope signing is provisional (`src/crypto.rs`): `FRX/0.3` + fields + sorted-key canonical JSON body. §10's signature canonicalization open issue is unsolved — never present this scheme as interoperable.
- Phase 1 only: no aggregates (Phase 2), no receipts/lineage/delegation (Phase 3). Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
- Relay verifies signatures, carries only `query` broadcasts, holds no history (queue drained on poll), and returns 429 + Retry-After under backpressure — never silent drops.
- Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits ordered results with honest `truncated`/`more_available` and no scores (I6).
- Index layout: Tantivy at `<data_dir>/index`, collections manifest at `<data_dir>/collections.toml`; `exposure` (metadata|full) gates whether `content` is returned.
- Egress checks live in the responder path (`src/node.rs` `respond`), not the relay — keep private collections unreachable there.
- Appendix B is executable policy: `tests/purges.rs` has one absence test per rejected mechanism (11 rows, not the RFC's informal "nine"). Add a negative test there before ever re-proposing one, and only if the rationale is addressed.
- `add`/`reindex` reset a collection (delete by manifest `name`) before re-adding, so deleted files don't linger; collection identity is its name, and same-named collections replace each other.
- Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale).
## Known gaps (Phase 2/3, intentional — don't fake them)
- No aggregate serving/dashboard (RFC §9 conformance is partial without it), no directory watching (new files need `reindex`), no receipts/lineage/delegation, no TLS, no consumer admission/relay discovery.
- Node query dedup is by `qid` only; there is no envelope replay/nonce window (RFC doesn't require one).
## Technical plans (deliberately not in the RFC) ## Technical plans (deliberately not in the RFC)
- Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface. - Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface.
- Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as enrichment members / backfill seeding. RFC §6 and Appendix A already name both as example derived corpora, so no new mechanisms are required; enrichment members are metadata-only exposure and sit outside the citation market. - Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as enrichment members / backfill seeding. RFC §6 and Appendix A already name both as example derived corpora, so no new mechanisms are required; enrichment members are metadata-only exposure and sit outside the citation market.
- Phase 1 remains the minimal two-node query/response demo (§7); the enrichment demo layers on top of it. - Phase 1 (two-node query/response) is built and tested; the enrichment demo layers on top of it.
- Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate. - Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate.
- frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher". - frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher".
- Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role.
- Matching accuracy is a project-health concern: start lexical (Tantivy), plan a hybrid cheap lexical gate + optional local embedding rerank (two-stage ingestion, Appendix A); embedding model stays local and replaceable (I2/I5). - Matching accuracy is a project-health concern: start lexical (Tantivy), plan a hybrid cheap lexical gate + optional local embedding rerank (two-stage ingestion, Appendix A); embedding model stays local and replaceable (I2/I5).
Generated
+3035
View File
File diff suppressed because it is too large Load Diff
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "frxd"
version = "0.1.0"
edition = "2024"
[dependencies]
anyhow = "1.0.104"
axum = "0.8.9"
chrono = { version = "0.4.45", default-features = false, features = ["clock", "std"] }
clap = { version = "4.6.7", features = ["derive"] }
ed25519-dalek = { version = "3.0.0", features = ["rand_core"] }
hex = "0.4.3"
rand = "0.8"
reqwest = { version = "0.13.5", default-features = false, features = ["json", "rustls", "http2"] }
serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151"
tantivy = "0.26.2"
tokio = { version = "1.53.1", features = ["full"] }
toml = "1.1.6"
[dev-dependencies]
tempfile = "3.27.0"
+55
View File
@@ -0,0 +1,55 @@
use std::path::PathBuf;
use anyhow::Result;
use clap::{Parser, Subcommand};
use frxd::commands;
#[derive(Parser)]
#[command(
name = "frx",
version,
about = "FRX client — local search and broadcast queries"
)]
struct Cli {
#[arg(long, global = true, default_value = "frxd.toml")]
config: PathBuf,
#[command(subcommand)]
command: Command,
}
#[derive(Subcommand)]
enum Command {
Search {
text: String,
#[arg(long, default_value_t = 10)]
limit: usize,
},
Query {
text: String,
#[arg(long)]
max_results: Option<usize>,
#[arg(long)]
timeout_ms: Option<u64>,
#[arg(long)]
local_only: bool,
},
Status,
}
#[tokio::main]
async fn main() -> Result<()> {
let cli = Cli::parse();
match cli.command {
Command::Search { text, limit } => commands::search(&cli.config, &text, limit).await?,
Command::Query {
text,
max_results,
timeout_ms,
local_only,
} => {
commands::query(&cli.config, &text, max_results, timeout_ms, !local_only).await?;
}
Command::Status => commands::status(&cli.config).await?,
}
Ok(())
}
+115
View File
@@ -0,0 +1,115 @@
use std::path::Path;
use anyhow::{Context, Result};
use crate::config::Config;
use crate::index::{Collection, LocalIndex, load_collections, save_collections};
use crate::message::{EXPOSURE_FULL, EXPOSURE_METADATA};
use crate::node;
use crate::render;
pub fn add(
config_path: &Path,
path: &Path,
name: Option<String>,
shared: bool,
exposure: &str,
) -> Result<()> {
let config = Config::load(config_path)?;
let index = LocalIndex::open(&config.index_dir())?;
let name = name.unwrap_or_else(|| {
path.file_name()
.and_then(|n| n.to_str())
.unwrap_or("collection")
.to_string()
});
let exposure = if exposure == EXPOSURE_FULL {
EXPOSURE_FULL
} else {
EXPOSURE_METADATA
};
let collection = Collection {
name: name.clone(),
path: path
.canonicalize()
.with_context(|| format!("resolving {}", path.display()))?
.display()
.to_string(),
shared,
exposure: exposure.to_string(),
};
let added = index.add_collection(&collection)?;
let manifest = config.collections_path();
let mut collections = load_collections(&manifest)?;
collections.retain(|existing| existing.name != name);
collections.push(collection);
save_collections(&manifest, &collections)?;
println!(
"indexed {added} file(s) from {} as collection '{name}' (shared={shared}, exposure={exposure})",
path.display()
);
Ok(())
}
pub fn reindex(config_path: &Path) -> Result<()> {
let config = Config::load(config_path)?;
let index = LocalIndex::open(&config.index_dir())?;
let collections = load_collections(&config.collections_path())?;
if collections.is_empty() {
println!("no collections registered; use `frxd add <path>`");
return Ok(());
}
for collection in collections {
let added = index.add_collection(&collection)?;
println!(
"collection '{}' ({}): {added} file(s)",
collection.name, collection.path
);
}
Ok(())
}
pub async fn search(config_path: &Path, text: &str, limit: usize) -> Result<()> {
let config = Config::load(config_path)?;
let index = LocalIndex::open(&config.index_dir())?;
let (hits, total) = index.search(text, limit, false)?;
render::local_hits(&hits, total);
Ok(())
}
pub async fn query(
config_path: &Path,
text: &str,
max_results: Option<usize>,
timeout_ms: Option<u64>,
network: bool,
) -> Result<()> {
let config = Config::load(config_path)?;
let base = format!("http://{}", config.node.listen);
let outcome = node::control_query(&base, text, max_results, timeout_ms, network).await?;
render::query_outcome(&outcome);
Ok(())
}
pub async fn status(config_path: &Path) -> Result<()> {
let config = Config::load(config_path)?;
let base = format!("http://{}", config.node.listen);
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(2))
.build()?;
match client.get(format!("{base}/v1/local/status")).send().await {
Ok(response) if response.status().is_success() => {
let value: serde_json::Value = response.json().await?;
println!("{}", serde_json::to_string_pretty(&value)?);
}
Ok(response) => println!("node at {base} responded {}", response.status()),
Err(_) => {
println!("node not running at {base}");
println!("config: {}", config_path.display());
println!("name: {}", config.node.name);
println!("data dir: {}", config.data_dir().display());
println!("relays: {:?}", config.node.relays);
}
}
Ok(())
}
+151
View File
@@ -0,0 +1,151 @@
use std::fs;
use std::path::{Path, PathBuf};
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use crate::crypto::Keypair;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Config {
pub node: NodeSection,
#[serde(default)]
pub query: QuerySection,
#[serde(default)]
pub index: IndexSection,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NodeSection {
pub name: String,
pub listen: String,
pub relays: Vec<String>,
#[serde(default)]
pub trusted_keys: Vec<String>,
#[serde(default = "default_true")]
pub responder: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct QuerySection {
#[serde(default = "default_max_results")]
pub max_results: usize,
#[serde(default = "default_timeout_ms")]
pub timeout_ms: u64,
}
impl Default for QuerySection {
fn default() -> Self {
Self {
max_results: default_max_results(),
timeout_ms: default_timeout_ms(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct IndexSection {
#[serde(default = "default_data_dir")]
pub data_dir: String,
}
impl Default for IndexSection {
fn default() -> Self {
Self {
data_dir: default_data_dir(),
}
}
}
fn default_true() -> bool {
true
}
fn default_max_results() -> usize {
5
}
fn default_timeout_ms() -> u64 {
2000
}
fn default_data_dir() -> String {
"./frx-data".to_string()
}
impl Config {
pub fn new(name: &str, listen: &str, relay: &str, data_dir: &str) -> Self {
Self {
node: NodeSection {
name: name.to_string(),
listen: listen.to_string(),
relays: vec![relay.to_string()],
trusted_keys: Vec::new(),
responder: true,
},
query: QuerySection::default(),
index: IndexSection {
data_dir: data_dir.to_string(),
},
}
}
pub fn load(path: &Path) -> Result<Self> {
let raw = fs::read_to_string(path)
.with_context(|| format!("reading config {}", path.display()))?;
toml::from_str(&raw).with_context(|| format!("parsing config {}", path.display()))
}
pub fn save(&self, path: &Path) -> Result<()> {
if let Some(parent) = path.parent() {
if !parent.as_os_str().is_empty() {
fs::create_dir_all(parent)?;
}
}
fs::write(path, toml::to_string_pretty(self)?)?;
Ok(())
}
pub fn data_dir(&self) -> PathBuf {
PathBuf::from(&self.index.data_dir)
}
pub fn index_dir(&self) -> PathBuf {
self.data_dir().join("index")
}
pub fn key_path(&self) -> PathBuf {
self.data_dir().join("key.hex")
}
pub fn collections_path(&self) -> PathBuf {
self.data_dir().join("collections.toml")
}
pub fn load_key(&self) -> Result<Keypair> {
let raw = fs::read_to_string(self.key_path())
.with_context(|| format!("reading key {}", self.key_path().display()))?;
Keypair::from_hex(&raw)
}
pub fn save_key(&self, key: &Keypair) -> Result<()> {
fs::create_dir_all(self.data_dir())?;
let path = self.key_path();
fs::write(&path, key.to_hex())?;
set_private_permissions(&path)?;
Ok(())
}
}
fn set_private_permissions(path: &Path) -> Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(path, fs::Permissions::from_mode(0o600))?;
}
#[cfg(not(unix))]
{
let _ = path;
}
Ok(())
}
+166
View File
@@ -0,0 +1,166 @@
use crate::PROTOCOL;
use crate::message::Envelope;
use anyhow::{Context, Result, anyhow};
use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey};
use rand::RngCore;
use serde_json::Value;
pub struct Keypair {
signing: SigningKey,
}
impl Keypair {
pub fn generate() -> Self {
let mut bytes = [0u8; 32];
rand::thread_rng().fill_bytes(&mut bytes);
Self {
signing: SigningKey::from_bytes(&bytes),
}
}
pub fn from_hex(s: &str) -> Result<Self> {
let raw = hex::decode(s.trim()).context("key is not hex")?;
let bytes: [u8; 32] = raw
.as_slice()
.try_into()
.map_err(|_| anyhow!("key must be 32 bytes"))?;
Ok(Self {
signing: SigningKey::from_bytes(&bytes),
})
}
pub fn to_hex(&self) -> String {
hex::encode(self.signing.to_bytes())
}
pub fn public_hex(&self) -> String {
hex::encode(self.signing.verifying_key().to_bytes())
}
pub fn verifying_key(&self) -> VerifyingKey {
self.signing.verifying_key()
}
pub fn sign(&self, bytes: &[u8]) -> String {
hex::encode(self.signing.sign(bytes).to_bytes())
}
}
pub fn random_nonce() -> String {
let mut bytes = [0u8; 16];
rand::thread_rng().fill_bytes(&mut bytes);
hex::encode(bytes)
}
pub fn random_id() -> String {
let mut bytes = [0u8; 8];
rand::thread_rng().fill_bytes(&mut bytes);
hex::encode(bytes)
}
pub fn canonical_json(value: &Value) -> String {
let mut out = String::new();
write_canonical(value, &mut out);
out
}
fn write_canonical(value: &Value, out: &mut String) {
match value {
Value::Null => out.push_str("null"),
Value::Bool(b) => out.push_str(if *b { "true" } else { "false" }),
Value::Number(n) => out.push_str(&n.to_string()),
Value::String(s) => out.push_str(&serde_json::to_string(s).expect("string serializes")),
Value::Array(items) => {
out.push('[');
for (i, item) in items.iter().enumerate() {
if i > 0 {
out.push(',');
}
write_canonical(item, out);
}
out.push(']');
}
Value::Object(map) => {
let mut keys: Vec<&String> = map.keys().collect();
keys.sort();
out.push('{');
for (i, key) in keys.iter().enumerate() {
if i > 0 {
out.push(',');
}
out.push_str(&serde_json::to_string(key).expect("key serializes"));
out.push(':');
write_canonical(&map[*key], out);
}
out.push('}');
}
}
}
pub fn signing_bytes(envelope: &Envelope) -> Vec<u8> {
format!(
"{}\n{}\n{}\n{}\n{}\n{}",
PROTOCOL,
envelope.msg_type,
envelope.from,
envelope.ts,
envelope.nonce,
canonical_json(&envelope.body)
)
.into_bytes()
}
pub fn verify_envelope(envelope: &Envelope) -> Result<()> {
let key_bytes = hex::decode(&envelope.from).context("from is not hex")?;
let key_bytes: [u8; 32] = key_bytes
.as_slice()
.try_into()
.map_err(|_| anyhow!("from must be a 32-byte ed25519 public key"))?;
let verifying_key =
VerifyingKey::from_bytes(&key_bytes).map_err(|e| anyhow!("bad public key: {e}"))?;
let sig_bytes = hex::decode(&envelope.sig).context("sig is not hex")?;
let sig_bytes: [u8; 64] = sig_bytes
.as_slice()
.try_into()
.map_err(|_| anyhow!("sig must be 64 bytes"))?;
let signature = Signature::from_bytes(&sig_bytes);
verifying_key
.verify_strict(&signing_bytes(envelope), &signature)
.map_err(|_| anyhow!("signature verification failed"))
}
pub fn now_ts() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn canonical_json_sorts_keys_recursively() {
let a = json!({"b": 1, "a": {"d": [3, 2], "c": "x"}});
let b = json!({"a": {"c": "x", "d": [3, 2]}, "b": 1});
assert_eq!(canonical_json(&a), canonical_json(&b));
assert_eq!(canonical_json(&a), r#"{"a":{"c":"x","d":[3,2]},"b":1}"#);
}
#[test]
fn envelope_sign_verify_roundtrip() {
let key = Keypair::generate();
let env = Envelope::new(&key, crate::message::TYPE_QUERY, json!({"text": "hello"}));
verify_envelope(&env).unwrap();
}
#[test]
fn tampered_body_fails_verification() {
let key = Keypair::generate();
let mut env = Envelope::new(&key, crate::message::TYPE_QUERY, json!({"text": "hello"}));
env.body = json!({"text": "hello", "extra": true});
assert!(verify_envelope(&env).is_err());
}
}
+242
View File
@@ -0,0 +1,242 @@
use std::fs;
use std::path::Path;
use std::time::SystemTime;
use chrono::{DateTime, SecondsFormat, Utc};
pub const MAX_BODY_BYTES: usize = 1_000_000;
pub struct Extracted {
pub title: String,
pub body: String,
pub published: String,
}
pub fn supported(path: &Path) -> bool {
match path.extension().and_then(|e| e.to_str()) {
Some(ext) => matches!(
ext.to_ascii_lowercase().as_str(),
"txt" | "md" | "markdown" | "html" | "htm" | "rst" | "log"
),
None => false,
}
}
pub fn extract_file(path: &Path) -> Option<Extracted> {
if !supported(path) {
return None;
}
let raw = fs::read_to_string(path).ok()?;
let ext = path
.extension()
.and_then(|e| e.to_str())
.unwrap_or("")
.to_ascii_lowercase();
let body = if ext == "html" || ext == "htm" {
strip_html(&raw)
} else {
collapse_blank_lines(&raw)
};
let body = truncate_chars(&body, MAX_BODY_BYTES);
let title = if ext == "html" || ext == "htm" {
html_title(&raw).unwrap_or_else(|| fallback_title(path, &body))
} else {
markdown_title(&raw).unwrap_or_else(|| fallback_title(path, &body))
};
let published = fs::metadata(path)
.and_then(|m| m.modified())
.ok()
.map(format_time)
.unwrap_or_default();
Some(Extracted {
title: truncate_chars(title.trim(), 200),
body,
published,
})
}
pub fn summary_of(body: &str) -> String {
truncate_chars(body.trim(), 300).replace('\n', " ")
}
fn fallback_title(path: &Path, body: &str) -> String {
if let Some(line) = body.lines().map(str::trim).find(|l| !l.is_empty()) {
if line.chars().count() > 3 {
return truncate_chars(line, 120);
}
}
path.file_stem()
.and_then(|s| s.to_str())
.unwrap_or("untitled")
.to_string()
}
fn markdown_title(raw: &str) -> Option<String> {
raw.lines()
.map(str::trim)
.find(|l| l.starts_with("# "))
.map(|l| l.trim_start_matches('#').trim().to_string())
}
fn html_title(raw: &str) -> Option<String> {
let lower = raw.to_ascii_lowercase();
let start = lower.find("<title")?;
let open_end = lower[start..].find('>')? + start + 1;
let end = lower[open_end..].find("</title>")? + open_end;
let title = decode_entities(raw[open_end..end].trim());
if title.is_empty() { None } else { Some(title) }
}
fn strip_html(raw: &str) -> String {
let mut out = String::with_capacity(raw.len() / 2);
let mut chars = raw.chars().peekable();
let mut skipping: Option<String> = None;
while let Some(c) = chars.next() {
if c == '<' {
let mut tag = String::new();
for t in chars.by_ref() {
if t == '>' {
break;
}
tag.push(t);
}
let name = tag
.trim_start_matches('/')
.trim()
.split(|c: char| c.is_whitespace() || c == '/')
.next()
.unwrap_or("")
.to_ascii_lowercase();
if skipping.is_none() && (name == "script" || name == "style") {
skipping = Some(name.clone());
} else if skipping.as_deref() == Some(name.as_str())
&& tag.trim_start().starts_with('/')
{
skipping = None;
}
out.push(' ');
continue;
}
if skipping.is_none() {
out.push(c);
}
}
collapse_whitespace(&decode_entities(&out))
}
fn collapse_whitespace(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut last_space = false;
for c in s.chars() {
if c.is_whitespace() {
if !last_space {
out.push(' ');
}
last_space = true;
} else {
out.push(c);
last_space = false;
}
}
out.trim().to_string()
}
fn collapse_blank_lines(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut blank_run = 0;
for line in s.lines() {
if line.trim().is_empty() {
blank_run += 1;
if blank_run > 1 {
continue;
}
} else {
blank_run = 0;
}
out.push_str(line);
out.push('\n');
}
out.trim().to_string()
}
fn decode_entities(s: &str) -> String {
s.replace("&nbsp;", " ")
.replace("&amp;", "&")
.replace("&lt;", "<")
.replace("&gt;", ">")
.replace("&quot;", "\"")
.replace("&#39;", "'")
.replace("&apos;", "'")
}
fn truncate_chars(s: &str, max: usize) -> String {
if s.len() <= max {
return s.to_string();
}
let end = s
.char_indices()
.take_while(|(i, _)| *i < max)
.last()
.map(|(i, c)| i + c.len_utf8())
.unwrap_or(0);
s[..end].to_string()
}
fn format_time(t: SystemTime) -> String {
let dt: DateTime<Utc> = t.into();
dt.to_rfc3339_opts(SecondsFormat::Secs, true)
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
#[test]
fn html_is_stripped_and_title_extracted() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("page.html");
let mut f = fs::File::create(&path).unwrap();
write!(
f,
"<html><head><title>Rust &amp; Ownership</title><style>p{{color:red}}</style></head><body><p>Hello <b>world</b></p><script>alert(1)</script></body></html>"
)
.unwrap();
let extracted = extract_file(&path).unwrap();
assert_eq!(extracted.title, "Rust & Ownership");
assert!(extracted.body.contains("Hello world"));
assert!(!extracted.body.contains("alert"));
assert!(!extracted.body.contains("color:red"));
}
#[test]
fn unsupported_extensions_are_ignored() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("archive.bin");
fs::write(&path, "binary-ish").unwrap();
assert!(!supported(&path));
assert!(extract_file(&path).is_none());
}
#[test]
fn oversized_bodies_are_truncated_on_char_boundaries() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("big.txt");
let mut body = "é".repeat(MAX_BODY_BYTES / 2 + 50);
body.push_str(" tail");
fs::write(&path, &body).unwrap();
let extracted = extract_file(&path).unwrap();
assert!(extracted.body.len() <= MAX_BODY_BYTES);
assert!(extracted.body.chars().all(|c| c == 'é'));
}
#[test]
fn markdown_heading_becomes_title() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("note.md");
fs::write(&path, "# Borrowing\n\nRules of borrowing.\n").unwrap();
let extracted = extract_file(&path).unwrap();
assert_eq!(extracted.title, "Borrowing");
assert!(extracted.body.contains("Rules of borrowing."));
}
}
+359
View File
@@ -0,0 +1,359 @@
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use tantivy::collector::{Count, TopDocs};
use tantivy::directory::MmapDirectory;
use tantivy::query::{AllQuery, BooleanQuery, Occur, Query, QueryParser, TermQuery};
use tantivy::schema::{Field, IndexRecordOption, STORED, STRING, Schema, TEXT, Value};
use tantivy::{Index, IndexReader, IndexWriter, TantivyDocument, Term, doc};
use crate::extract::{extract_file, summary_of, supported};
use crate::message::{EXPOSURE_FULL, ResponseItem};
const WRITER_BUDGET: usize = 50_000_000;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Collection {
pub name: String,
pub path: String,
pub shared: bool,
pub exposure: String,
}
#[derive(Debug, Default, Serialize, Deserialize)]
struct CollectionsFile {
#[serde(default, rename = "collection")]
collections: Vec<Collection>,
}
pub fn load_collections(manifest: &Path) -> Result<Vec<Collection>> {
if !manifest.exists() {
return Ok(Vec::new());
}
let raw = fs::read_to_string(manifest).context("reading collections manifest")?;
let parsed: CollectionsFile = toml::from_str(&raw).context("parsing collections manifest")?;
Ok(parsed.collections)
}
pub fn save_collections(manifest: &Path, collections: &[Collection]) -> Result<()> {
if let Some(parent) = manifest.parent() {
fs::create_dir_all(parent)?;
}
let file = CollectionsFile {
collections: collections.to_vec(),
};
fs::write(manifest, toml::to_string_pretty(&file)?)?;
Ok(())
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SearchHit {
pub url: String,
pub title: String,
pub summary: String,
pub published: String,
pub exposure: String,
pub collection: String,
pub shared: bool,
pub content: Option<String>,
}
pub fn response_items(hits: &[SearchHit]) -> Vec<ResponseItem> {
hits.iter()
.map(|hit| ResponseItem {
url: hit.url.clone(),
title: hit.title.clone(),
summary: hit.summary.clone(),
published: hit.published.clone(),
exposure: hit.exposure.clone(),
content: if hit.exposure == EXPOSURE_FULL {
hit.content.clone()
} else {
None
},
})
.collect()
}
struct Fields {
url: Field,
title: Field,
body: Field,
summary: Field,
published: Field,
exposure: Field,
collection: Field,
shared: Field,
}
pub struct LocalIndex {
index: Index,
reader: IndexReader,
writer: Mutex<IndexWriter>,
fields: Fields,
}
impl LocalIndex {
pub fn open(dir: &Path) -> Result<Self> {
fs::create_dir_all(dir)?;
let schema = build_schema();
let index = Index::open_or_create(MmapDirectory::open(dir)?, schema)
.context("opening tantivy index")?;
let schema = index.schema();
let fields = Fields {
url: schema.get_field("url")?,
title: schema.get_field("title")?,
body: schema.get_field("body")?,
summary: schema.get_field("summary")?,
published: schema.get_field("published")?,
exposure: schema.get_field("exposure")?,
collection: schema.get_field("collection")?,
shared: schema.get_field("shared")?,
};
let reader = index.reader()?;
let writer = index.writer::<TantivyDocument>(WRITER_BUDGET)?;
Ok(Self {
index,
reader,
writer: Mutex::new(writer),
fields,
})
}
pub fn add_collection(&self, collection: &Collection) -> Result<usize> {
let root = PathBuf::from(&collection.path);
{
let writer = self.writer.lock().expect("writer lock");
writer.delete_term(Term::from_field_text(
self.fields.collection,
&collection.name,
));
}
let mut files = Vec::new();
walk(&root, &mut files);
let mut added = 0;
for file in files {
if self.add_file(collection, &file)? {
added += 1;
}
}
self.commit()?;
Ok(added)
}
pub fn add_file(&self, collection: &Collection, path: &Path) -> Result<bool> {
let Some(extracted) = extract_file(path) else {
return Ok(false);
};
let canonical = path.canonicalize().unwrap_or_else(|_| path.to_path_buf());
let url = format!("file://{}", canonical.display());
let summary = summary_of(&extracted.body);
let shared = if collection.shared { "true" } else { "false" };
let writer = self.writer.lock().expect("writer lock");
writer.delete_term(Term::from_field_text(self.fields.url, &url));
writer.add_document(doc!(
self.fields.url => url.as_str(),
self.fields.title => extracted.title.as_str(),
self.fields.body => extracted.body.as_str(),
self.fields.summary => summary.as_str(),
self.fields.published => extracted.published.as_str(),
self.fields.exposure => collection.exposure.as_str(),
self.fields.collection => collection.name.as_str(),
self.fields.shared => shared,
))?;
Ok(true)
}
pub fn commit(&self) -> Result<()> {
self.writer.lock().expect("writer lock").commit()?;
self.reader.reload()?;
Ok(())
}
pub fn search(
&self,
text: &str,
limit: usize,
only_shared: bool,
) -> Result<(Vec<SearchHit>, u64)> {
let limit = limit.max(1);
self.reader.reload()?;
let searcher = self.reader.searcher();
let user_query: Box<dyn Query> = if text.trim().is_empty() {
Box::new(AllQuery)
} else {
let parser =
QueryParser::for_index(&self.index, vec![self.fields.title, self.fields.body]);
let (query, _errors) = parser.parse_query_lenient(text);
query
};
let query: Box<dyn Query> = if only_shared {
let shared_term = TermQuery::new(
Term::from_field_text(self.fields.shared, "true"),
IndexRecordOption::Basic,
);
Box::new(BooleanQuery::new(vec![
(Occur::Must, user_query),
(Occur::Must, Box::new(shared_term)),
]))
} else {
user_query
};
let total = searcher.search(&*query, &Count)? as u64;
let top = searcher.search(&*query, &TopDocs::with_limit(limit).order_by_score())?;
let mut hits = Vec::with_capacity(top.len());
for (_score, address) in top {
let document: TantivyDocument = searcher.doc(address)?;
hits.push(SearchHit {
url: text_value(&document, self.fields.url),
title: text_value(&document, self.fields.title),
summary: text_value(&document, self.fields.summary),
published: text_value(&document, self.fields.published),
exposure: text_value(&document, self.fields.exposure),
collection: text_value(&document, self.fields.collection),
shared: text_value(&document, self.fields.shared) == "true",
content: Some(text_value(&document, self.fields.body)),
});
}
Ok((hits, total))
}
pub fn doc_count(&self) -> u64 {
self.reader.searcher().num_docs()
}
}
fn text_value(document: &TantivyDocument, field: Field) -> String {
document
.get_first(field)
.and_then(|value| value.as_str())
.unwrap_or("")
.to_string()
}
fn build_schema() -> Schema {
let mut builder = Schema::builder();
builder.add_text_field("url", STRING | STORED);
builder.add_text_field("title", TEXT | STORED);
builder.add_text_field("body", TEXT | STORED);
builder.add_text_field("summary", STORED);
builder.add_text_field("published", STORED);
builder.add_text_field("exposure", STRING | STORED);
builder.add_text_field("collection", STRING | STORED);
builder.add_text_field("shared", STRING | STORED);
builder.build()
}
fn walk(root: &Path, out: &mut Vec<PathBuf>) {
let Ok(entries) = fs::read_dir(root) else {
return;
};
for entry in entries.flatten() {
let Ok(file_type) = entry.file_type() else {
continue;
};
if file_type.is_symlink() {
continue;
}
let name = entry.file_name();
let name = name.to_string_lossy();
if file_type.is_dir() {
if name.starts_with('.') {
continue;
}
walk(&entry.path(), out);
} else if file_type.is_file() && supported(&entry.path()) {
out.push(entry.path());
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::message::EXPOSURE_METADATA;
fn collection(name: &str, root: &Path, shared: bool, exposure: &str) -> Collection {
Collection {
name: name.to_string(),
path: root.display().to_string(),
shared,
exposure: exposure.to_string(),
}
}
#[test]
fn shared_filter_and_exposure_are_enforced() {
let temp = tempfile::tempdir().unwrap();
let shared_dir = temp.path().join("shared");
let private_dir = temp.path().join("private");
fs::create_dir_all(&shared_dir).unwrap();
fs::create_dir_all(&private_dir).unwrap();
fs::write(shared_dir.join("a.txt"), "rust ownership rules").unwrap();
fs::write(private_dir.join("b.txt"), "rust secret notes").unwrap();
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
index
.add_collection(&collection("shared", &shared_dir, true, EXPOSURE_FULL))
.unwrap();
index
.add_collection(&collection(
"private",
&private_dir,
false,
EXPOSURE_METADATA,
))
.unwrap();
let (hits, total) = index.search("rust", 10, false).unwrap();
assert_eq!(total, 2);
assert_eq!(hits.len(), 2);
let (shared_hits, shared_total) = index.search("rust", 10, true).unwrap();
assert_eq!(shared_total, 1);
assert_eq!(shared_hits.len(), 1);
assert!(shared_hits[0].url.contains("shared"));
let items = response_items(&shared_hits);
assert!(items[0].content.is_some());
}
#[test]
fn reindex_removes_deleted_files() {
let temp = tempfile::tempdir().unwrap();
let dir = temp.path().join("corpus");
fs::create_dir_all(&dir).unwrap();
fs::write(dir.join("a.txt"), "rust one").unwrap();
fs::write(dir.join("b.txt"), "rust two").unwrap();
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
index
.add_collection(&collection("test", &dir, true, EXPOSURE_FULL))
.unwrap();
assert_eq!(index.search("rust", 10, false).unwrap().1, 2);
fs::remove_file(dir.join("b.txt")).unwrap();
index
.add_collection(&collection("test", &dir, true, EXPOSURE_FULL))
.unwrap();
assert_eq!(index.search("rust", 10, false).unwrap().1, 1);
}
#[test]
fn metadata_exposure_withholds_content() {
let temp = tempfile::tempdir().unwrap();
let dir = temp.path().join("corpus");
fs::create_dir_all(&dir).unwrap();
fs::write(dir.join("a.txt"), "metadata only").unwrap();
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
index
.add_collection(&collection("test", &dir, true, EXPOSURE_METADATA))
.unwrap();
let (hits, _) = index.search("metadata", 10, true).unwrap();
let items = response_items(&hits);
assert!(items[0].content.is_none());
assert_eq!(items[0].exposure, EXPOSURE_METADATA);
}
}
+12
View File
@@ -0,0 +1,12 @@
pub mod commands;
pub mod config;
pub mod crypto;
pub mod extract;
pub mod index;
pub mod message;
pub mod node;
pub mod relay;
pub mod render;
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
pub const PROTOCOL: &str = "FRX/0.3";
+141
View File
@@ -0,0 +1,141 @@
use std::path::PathBuf;
use anyhow::{Result, bail};
use clap::{Parser, Subcommand, ValueEnum};
use frxd::config::Config;
use frxd::crypto::Keypair;
use frxd::message::{EXPOSURE_FULL, EXPOSURE_METADATA};
use frxd::{commands, node, relay};
#[derive(Parser)]
#[command(
name = "frxd",
version,
about = "FRX member node — querier, responder, and local index (Draft 0.3)"
)]
struct Cli {
#[arg(long, global = true, default_value = "frxd.toml")]
config: PathBuf,
#[command(subcommand)]
command: Command,
}
#[derive(Subcommand)]
enum Command {
Init {
#[arg(long, default_value = "member")]
name: String,
#[arg(long, default_value = "127.0.0.1:7701")]
listen: String,
#[arg(long, default_value = "http://127.0.0.1:7700")]
relay: String,
#[arg(long, default_value = "./frx-data")]
data_dir: String,
#[arg(long)]
force: bool,
},
Add {
path: PathBuf,
#[arg(long)]
name: Option<String>,
#[arg(long)]
shared: bool,
#[arg(long, value_enum, default_value_t = Exposure::Full)]
exposure: Exposure,
},
Reindex,
Search {
text: String,
#[arg(long, default_value_t = 10)]
limit: usize,
},
Query {
text: String,
#[arg(long)]
max_results: Option<usize>,
#[arg(long)]
timeout_ms: Option<u64>,
#[arg(long)]
local_only: bool,
},
Serve,
Relay {
#[arg(long, default_value = "127.0.0.1:7700")]
listen: String,
#[arg(long, default_value_t = relay::DEFAULT_CAPACITY)]
capacity: usize,
},
Status,
}
#[derive(Clone, Copy, ValueEnum)]
enum Exposure {
Metadata,
Full,
}
#[tokio::main]
async fn main() -> Result<()> {
let cli = Cli::parse();
match cli.command {
Command::Init {
name,
listen,
relay,
data_dir,
force,
} => {
if cli.config.exists() && !force {
bail!(
"config {} already exists (use --force to overwrite)",
cli.config.display()
);
}
let config = Config::new(&name, &listen, &relay, &data_dir);
let key = Keypair::generate();
config.save_key(&key)?;
config.save(&cli.config)?;
println!("wrote config {}", cli.config.display());
println!("wrote key {}", config.key_path().display());
println!("pubkey {}", key.public_hex());
println!("data dir {}", config.data_dir().display());
}
Command::Add {
path,
name,
shared,
exposure,
} => {
let exposure = match exposure {
Exposure::Metadata => EXPOSURE_METADATA,
Exposure::Full => EXPOSURE_FULL,
};
commands::add(&cli.config, &path, name, shared, exposure)?;
}
Command::Reindex => commands::reindex(&cli.config)?,
Command::Search { text, limit } => commands::search(&cli.config, &text, limit).await?,
Command::Query {
text,
max_results,
timeout_ms,
local_only,
} => {
commands::query(&cli.config, &text, max_results, timeout_ms, !local_only).await?;
}
Command::Serve => {
let config = Config::load(&cli.config)?;
let handle = node::Node::start(config).await?;
println!("frxd listening on http://{}", handle.addr);
println!("pubkey {}", handle.pubkey);
println!("control API POST http://{}/v1/local/query", handle.addr);
tokio::signal::ctrl_c().await?;
}
Command::Relay { listen, capacity } => {
let (listener, addr) = relay::bind(&listen).await?;
println!("relay listening on http://{addr} (capacity {capacity})");
relay::run(listener, capacity).await?;
}
Command::Status => commands::status(&cli.config).await?,
}
Ok(())
}
+184
View File
@@ -0,0 +1,184 @@
use crate::crypto::{Keypair, now_ts, random_nonce, signing_bytes, verify_envelope};
use anyhow::{Context, Result, anyhow};
use serde::de::DeserializeOwned;
use serde::{Deserialize, Serialize};
use serde_json::Value;
pub const TYPE_QUERY: &str = "query";
pub const TYPE_RESPONSE: &str = "response";
pub const TYPE_AGGREGATE: &str = "aggregate";
pub const EXPOSURE_METADATA: &str = "metadata";
pub const EXPOSURE_FULL: &str = "full";
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Envelope {
#[serde(rename = "type")]
pub msg_type: String,
pub from: String,
pub ts: u64,
pub nonce: String,
pub body: Value,
pub sig: String,
}
impl Envelope {
pub fn new(key: &Keypair, msg_type: &str, body: Value) -> Self {
let mut envelope = Self {
msg_type: msg_type.to_string(),
from: key.public_hex(),
ts: now_ts(),
nonce: random_nonce(),
body,
sig: String::new(),
};
envelope.sig = key.sign(&signing_bytes(&envelope));
envelope
}
pub fn verify(&self) -> Result<()> {
verify_envelope(self)
}
pub fn parse_body<T: DeserializeOwned>(&self) -> Result<T> {
serde_json::from_value(self.body.clone()).context("malformed body")
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Budget {
pub max_results: usize,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct QueryBody {
pub qid: String,
pub text: String,
#[serde(default)]
pub entities: Vec<String>,
pub budget: Budget,
}
impl QueryBody {
pub fn new(text: &str, max_results: usize) -> Self {
Self {
qid: crate::crypto::random_id(),
text: text.to_string(),
entities: Vec::new(),
budget: Budget {
max_results: max_results.clamp(1, 1000),
},
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResponseItem {
pub url: String,
pub title: String,
pub summary: String,
pub published: String,
pub exposure: String,
pub content: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResponseBody {
pub qid: String,
pub results: Vec<ResponseItem>,
pub truncated: bool,
pub more_available: u64,
pub cursor: Option<String>,
}
pub fn build_response(
qid: &str,
hits: Vec<ResponseItem>,
total: u64,
max_results: usize,
) -> ResponseBody {
let max_results = max_results.max(1);
let mut results = hits;
results.truncate(max_results);
let more_available = total.saturating_sub(results.len() as u64);
ResponseBody {
qid: qid.to_string(),
results,
truncated: more_available > 0,
more_available,
cursor: None,
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AggregateBody {
pub period: String,
pub sent: u64,
pub passed: u64,
pub cited: u64,
}
pub fn require_type(envelope: &Envelope, expected: &str) -> Result<()> {
if envelope.msg_type != expected {
return Err(anyhow!(
"expected message type {expected}, got {}",
envelope.msg_type
));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn item(n: usize) -> ResponseItem {
ResponseItem {
url: format!("file:///doc{n}.txt"),
title: format!("doc {n}"),
summary: String::new(),
published: String::new(),
exposure: EXPOSURE_METADATA.to_string(),
content: None,
}
}
#[test]
fn max_results_is_respected() {
let response = build_response("q1", vec![item(1), item(2), item(3)], 10, 2);
assert_eq!(response.results.len(), 2);
}
#[test]
fn truncation_is_honest() {
let response = build_response("q1", vec![item(1), item(2), item(3)], 10, 2);
assert!(response.truncated);
assert_eq!(response.more_available, 8);
let response = build_response("q2", vec![item(1), item(2)], 2, 5);
assert!(!response.truncated);
assert_eq!(response.more_available, 0);
}
#[test]
fn response_carries_no_scores() {
let response = build_response("q1", vec![item(1)], 1, 5);
let json = serde_json::to_string(&response).unwrap();
assert!(!json.contains("score"));
assert!(!json.contains("relevance"));
}
#[test]
fn zero_budget_is_clamped() {
let query = QueryBody::new("anything", 0);
assert_eq!(query.budget.max_results, 1);
}
#[test]
fn envelope_rejects_wrong_type() {
let key = Keypair::generate();
let envelope = Envelope::new(&key, TYPE_RESPONSE, json!({}));
assert!(require_type(&envelope, TYPE_QUERY).is_err());
}
}
+434
View File
@@ -0,0 +1,434 @@
use std::collections::{HashMap, HashSet};
use std::net::SocketAddr;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use anyhow::{Context, Result, anyhow};
use axum::extract::State;
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use axum::routing::{get, post};
use axum::{Json, Router};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use tokio::net::TcpListener;
use tokio::task::JoinHandle;
use crate::config::Config;
use crate::crypto::Keypair;
use crate::index::{LocalIndex, SearchHit, response_items};
use crate::message::{
Envelope, QueryBody, ResponseBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE,
build_response,
};
pub struct Node {
pub config: Config,
pub key: Keypair,
index: LocalIndex,
seen: Mutex<HashSet<String>>,
pending: Mutex<HashMap<String, Vec<(String, ResponseBody)>>>,
client: reqwest::Client,
sent: AtomicU64,
received: AtomicU64,
}
pub struct NodeHandle {
pub addr: SocketAddr,
pub pubkey: String,
pub node: Arc<Node>,
tasks: Vec<JoinHandle<()>>,
}
impl Drop for NodeHandle {
fn drop(&mut self) {
for task in &self.tasks {
task.abort();
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RemoteResponse {
pub member: String,
pub results: Vec<ResponseItem>,
pub truncated: bool,
pub more_available: u64,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MergedItem {
pub provenance: String,
#[serde(flatten)]
pub item: ResponseItem,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LocalQueryOutcome {
pub qid: String,
pub text: String,
pub local: LocalPart,
pub responses: Vec<RemoteResponse>,
pub merged: Vec<MergedItem>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LocalPart {
pub results: Vec<ResponseItem>,
pub total: u64,
}
impl Node {
pub fn open(config: Config) -> Result<Arc<Self>> {
let key = config.load_key()?;
let index = LocalIndex::open(&config.index_dir())?;
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(15))
.build()
.context("building http client")?;
Ok(Arc::new(Self {
config,
key,
index,
seen: Mutex::new(HashSet::new()),
pending: Mutex::new(HashMap::new()),
client,
sent: AtomicU64::new(0),
received: AtomicU64::new(0),
}))
}
pub fn doc_count(&self) -> u64 {
self.index.doc_count()
}
pub fn sent(&self) -> u64 {
self.sent.load(Ordering::SeqCst)
}
pub fn received(&self) -> u64 {
self.received.load(Ordering::SeqCst)
}
pub fn local_search(&self, text: &str, limit: usize) -> Result<(Vec<SearchHit>, u64)> {
self.index.search(text, limit, false)
}
pub async fn start(config: Config) -> Result<NodeHandle> {
let node = Node::open(config)?;
let listener = TcpListener::bind(&node.config.node.listen)
.await
.with_context(|| format!("binding {}", node.config.node.listen))?;
let addr = listener.local_addr()?;
let mut tasks = Vec::new();
for relay in node.config.node.relays.clone() {
tasks.push(tokio::spawn(poll_relay(node.clone(), relay)));
}
let app = router(node.clone());
tasks.push(tokio::spawn(async move {
if let Err(error) = axum::serve(listener, app).await {
eprintln!("control server stopped: {error}");
}
}));
Ok(NodeHandle {
addr,
pubkey: node.key.public_hex(),
node,
tasks,
})
}
pub async fn local_query(
&self,
text: &str,
max_results: Option<usize>,
timeout_ms: Option<u64>,
network: bool,
) -> Result<LocalQueryOutcome> {
let max = max_results.unwrap_or(self.config.query.max_results).max(1);
let query = QueryBody::new(text, max);
let qid = query.qid.clone();
let (local_hits, local_total) = self.index.search(text, max, false)?;
let local_items = response_items(&local_hits);
let mut responses = Vec::new();
if network {
self.pending
.lock()
.expect("pending lock")
.insert(qid.clone(), Vec::new());
self.sent.fetch_add(1, Ordering::SeqCst);
let envelope = Envelope::new(&self.key, TYPE_QUERY, serde_json::to_value(&query)?);
self.publish(&envelope).await;
let timeout = Duration::from_millis(timeout_ms.unwrap_or(self.config.query.timeout_ms));
let deadline = Instant::now() + timeout;
while Instant::now() < deadline {
tokio::time::sleep(Duration::from_millis(25)).await;
}
let collected = self
.pending
.lock()
.expect("pending lock")
.remove(&qid)
.unwrap_or_default();
self.received
.fetch_add(collected.len() as u64, Ordering::SeqCst);
for (member, body) in collected {
responses.push(RemoteResponse {
member,
results: body.results,
truncated: body.truncated,
more_available: body.more_available,
});
}
}
let mut merged = Vec::new();
let mut seen_urls = HashSet::new();
for item in &local_items {
if seen_urls.insert(item.url.clone()) {
merged.push(MergedItem {
provenance: "local".to_string(),
item: item.clone(),
});
}
}
for response in &responses {
for item in &response.results {
if seen_urls.insert(item.url.clone()) {
merged.push(MergedItem {
provenance: response.member.clone(),
item: item.clone(),
});
}
}
}
Ok(LocalQueryOutcome {
qid,
text: text.to_string(),
local: LocalPart {
results: local_items,
total: local_total,
},
responses,
merged,
})
}
async fn publish(&self, envelope: &Envelope) -> usize {
let mut delivered = 0;
for relay in &self.config.node.relays {
let url = format!("{}/v1/publish", relay.trim_end_matches('/'));
match self.client.post(&url).json(envelope).send().await {
Ok(response) if response.status().is_success() => delivered += 1,
Ok(response) => eprintln!("relay {relay} rejected query: {}", response.status()),
Err(error) => eprintln!("relay {relay} unreachable: {error}"),
}
}
delivered
}
async fn dispatch(self: &Arc<Self>, envelope: Envelope, relay: &str) {
if envelope.verify().is_err() {
return;
}
if !self.config.node.trusted_keys.is_empty()
&& !self.config.node.trusted_keys.contains(&envelope.from)
{
return;
}
match envelope.msg_type.as_str() {
TYPE_QUERY => {
if envelope.from == self.key.public_hex() || !self.config.node.responder {
return;
}
let Ok(query) = envelope.parse_body::<QueryBody>() else {
return;
};
if query.text.trim().is_empty() || query.qid.is_empty() {
return;
}
{
let mut seen = self.seen.lock().expect("seen lock");
if !seen.insert(query.qid.clone()) {
return;
}
if seen.len() > 10_000 {
seen.clear();
}
}
let node = self.clone();
let relay = relay.to_string();
let querier = envelope.from.clone();
tokio::spawn(async move {
if let Err(error) = node.respond(&query, &querier, &relay).await {
eprintln!("responder failed: {error}");
}
});
}
TYPE_RESPONSE => {
let Ok(body) = envelope.parse_body::<ResponseBody>() else {
return;
};
let mut pending = self.pending.lock().expect("pending lock");
if let Some(list) = pending.get_mut(&body.qid) {
list.push((envelope.from.clone(), body));
}
}
TYPE_AGGREGATE => {}
_ => {}
}
}
async fn respond(&self, query: &QueryBody, querier: &str, relay: &str) -> Result<()> {
let max = query.budget.max_results.clamp(1, 1000);
let (hits, total) = self.index.search(&query.text, max, true)?;
if hits.is_empty() {
return Ok(());
}
let body = build_response(&query.qid, response_items(&hits), total, max);
let envelope = Envelope::new(&self.key, TYPE_RESPONSE, serde_json::to_value(&body)?);
let mut relays = vec![relay.to_string()];
for configured in &self.config.node.relays {
if !relays.contains(configured) {
relays.push(configured.clone());
}
}
let mut last_error: Option<anyhow::Error> = None;
for candidate in relays {
let url = format!(
"{}/v1/unicast?to={}",
candidate.trim_end_matches('/'),
querier
);
match self.client.post(&url).json(&envelope).send().await {
Ok(response) if response.status().is_success() => return Ok(()),
Ok(response) => {
last_error = Some(anyhow!(
"relay {candidate} rejected response: {}",
response.status()
))
}
Err(error) => last_error = Some(anyhow!("relay {candidate} unreachable: {error}")),
}
}
Err(last_error.unwrap_or_else(|| anyhow!("no relays configured")))
}
}
async fn poll_relay(node: Arc<Node>, relay: String) {
let base = relay.trim_end_matches('/').to_string();
loop {
let url = format!(
"{}/v1/poll?member={}&timeout_ms=20000",
base,
node.key.public_hex()
);
match node.client.get(&url).send().await {
Ok(response) if response.status() == reqwest::StatusCode::NO_CONTENT => continue,
Ok(response) if response.status().is_success() => {
match response.json::<Value>().await {
Ok(payload) => {
if let Some(messages) = payload.get("messages").and_then(Value::as_array) {
for message in messages {
if let Ok(envelope) =
serde_json::from_value::<Envelope>(message.clone())
{
node.dispatch(envelope, &base).await;
}
}
}
}
Err(_) => tokio::time::sleep(Duration::from_secs(1)).await,
}
}
Ok(_) => tokio::time::sleep(Duration::from_secs(1)).await,
Err(_) => tokio::time::sleep(Duration::from_secs(1)).await,
}
}
}
pub fn router(node: Arc<Node>) -> Router {
Router::new()
.route("/v1/local/query", post(local_query))
.route("/v1/local/status", get(local_status))
.with_state(node)
}
#[derive(Deserialize)]
pub struct LocalQueryRequest {
pub text: String,
pub max_results: Option<usize>,
pub timeout_ms: Option<u64>,
#[serde(default = "default_network")]
pub network: bool,
}
fn default_network() -> bool {
true
}
async fn local_query(
State(node): State<Arc<Node>>,
Json(request): Json<LocalQueryRequest>,
) -> Response {
match node
.local_query(
&request.text,
request.max_results,
request.timeout_ms,
request.network,
)
.await
{
Ok(outcome) => (StatusCode::OK, Json(outcome)).into_response(),
Err(error) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({ "error": error.to_string() })),
)
.into_response(),
}
}
async fn local_status(State(node): State<Arc<Node>>) -> Response {
Json(json!({
"name": node.config.node.name,
"pubkey": node.key.public_hex(),
"listen": node.config.node.listen,
"relays": node.config.node.relays,
"responder": node.config.node.responder,
"doc_count": node.doc_count(),
"sent": node.sent(),
"received": node.received(),
}))
.into_response()
}
pub async fn control_query(
base: &str,
text: &str,
max_results: Option<usize>,
timeout_ms: Option<u64>,
network: bool,
) -> Result<Value> {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.build()?;
let url = format!("{}/v1/local/query", base.trim_end_matches('/'));
let response = client
.post(&url)
.json(&json!({
"text": text,
"max_results": max_results,
"timeout_ms": timeout_ms,
"network": network,
}))
.send()
.await
.with_context(|| format!("calling local node at {url} (is `frxd serve` running?)"))?;
let status = response.status();
let value: Value = response.json().await.context("parsing node response")?;
if !status.is_success() {
return Err(anyhow!("local node error {status}: {value}"));
}
Ok(value)
}
+191
View File
@@ -0,0 +1,191 @@
use std::collections::{HashMap, VecDeque};
use std::net::SocketAddr;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use anyhow::Result;
use axum::extract::{Query, State};
use axum::http::{StatusCode, header};
use axum::response::{IntoResponse, Response};
use axum::routing::{get, post};
use axum::{Json, Router};
use serde::Deserialize;
use serde_json::json;
use tokio::net::TcpListener;
use crate::message::{Envelope, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE};
pub const DEFAULT_CAPACITY: usize = 256;
#[derive(Default)]
struct MemberQueue {
items: VecDeque<(u64, Envelope)>,
}
struct Inner {
members: HashMap<String, MemberQueue>,
seq: u64,
}
pub struct Relay {
inner: Mutex<Inner>,
capacity: usize,
}
impl Relay {
pub fn new(capacity: usize) -> Arc<Self> {
Arc::new(Self {
inner: Mutex::new(Inner {
members: HashMap::new(),
seq: 0,
}),
capacity,
})
}
fn push(&self, targets: Option<&str>, envelope: Envelope) -> Result<usize, StatusCode> {
let mut inner = self.inner.lock().expect("relay lock");
if inner
.members
.values()
.any(|q| q.items.len() >= self.capacity)
{
return Err(StatusCode::TOO_MANY_REQUESTS);
}
inner.seq += 1;
let seq = inner.seq;
match targets {
Some(member) => {
let Some(queue) = inner.members.get_mut(member) else {
return Err(StatusCode::NOT_FOUND);
};
queue.items.push_back((seq, envelope));
Ok(1)
}
None => {
let from = envelope.from.clone();
inner.members.entry(from).or_default();
let mut delivered = 0;
for queue in inner.members.values_mut() {
queue.items.push_back((seq, envelope.clone()));
delivered += 1;
}
Ok(delivered)
}
}
}
}
pub fn router(relay: Arc<Relay>) -> Router {
Router::new()
.route("/health", get(health))
.route("/v1/publish", post(publish))
.route("/v1/unicast", post(unicast))
.route("/v1/poll", get(poll))
.with_state(relay)
}
pub async fn run(listener: TcpListener, capacity: usize) -> Result<()> {
let relay = Relay::new(capacity);
axum::serve(listener, router(relay)).await?;
Ok(())
}
pub async fn bind(addr: &str) -> Result<(TcpListener, SocketAddr)> {
let listener = TcpListener::bind(addr).await?;
let local = listener.local_addr()?;
Ok((listener, local))
}
async fn health() -> &'static str {
"ok"
}
async fn publish(State(relay): State<Arc<Relay>>, Json(envelope): Json<Envelope>) -> Response {
if envelope.verify().is_err() {
return bad_request("invalid signature");
}
if envelope.msg_type != TYPE_QUERY {
return bad_request("relay carries broadcast queries only");
}
match relay.push(None, envelope) {
Ok(delivered) => (
StatusCode::ACCEPTED,
Json(json!({ "delivered": delivered })),
)
.into_response(),
Err(status) => backpressure(status),
}
}
#[derive(Deserialize)]
struct UnicastParams {
to: String,
}
async fn unicast(
State(relay): State<Arc<Relay>>,
Query(params): Query<UnicastParams>,
Json(envelope): Json<Envelope>,
) -> Response {
if envelope.verify().is_err() {
return bad_request("invalid signature");
}
if envelope.msg_type != TYPE_RESPONSE && envelope.msg_type != TYPE_AGGREGATE {
return bad_request("unicast carries responses and aggregates only");
}
match relay.push(Some(&params.to), envelope) {
Ok(_) => (StatusCode::OK, Json(json!({ "delivered": true }))).into_response(),
Err(StatusCode::NOT_FOUND) => (
StatusCode::NOT_FOUND,
Json(json!({ "error": "member not connected" })),
)
.into_response(),
Err(status) => backpressure(status),
}
}
#[derive(Deserialize)]
struct PollParams {
member: String,
timeout_ms: Option<u64>,
}
async fn poll(State(relay): State<Arc<Relay>>, Query(params): Query<PollParams>) -> Response {
if params.member.is_empty() || hex::decode(&params.member).is_err() {
return bad_request("valid member key required");
}
let timeout = Duration::from_millis(params.timeout_ms.unwrap_or(25_000).min(60_000));
let deadline = Instant::now() + timeout;
loop {
let batch: Vec<Envelope> = {
let mut inner = relay.inner.lock().expect("relay lock");
let queue = inner.members.entry(params.member.clone()).or_default();
queue
.items
.drain(..)
.map(|(_, envelope)| envelope)
.collect()
};
if !batch.is_empty() {
return (StatusCode::OK, Json(json!({ "messages": batch }))).into_response();
}
if Instant::now() >= deadline {
return StatusCode::NO_CONTENT.into_response();
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
fn bad_request(message: &str) -> Response {
(StatusCode::BAD_REQUEST, Json(json!({ "error": message }))).into_response()
}
fn backpressure(status: StatusCode) -> Response {
(
status,
[(header::RETRY_AFTER, "1")],
Json(json!({ "error": "transport backpressure" })),
)
.into_response()
}
+63
View File
@@ -0,0 +1,63 @@
use serde_json::Value;
use crate::index::SearchHit;
pub fn local_hits(hits: &[SearchHit], total: u64) {
println!("{total} local match(es), showing {}", hits.len());
for hit in hits {
let scope = if hit.shared { "shared" } else { "private" };
println!("- [{} / {}] {}", hit.collection, scope, hit.title);
println!(" {}", hit.url);
if !hit.summary.is_empty() {
println!(" {}", hit.summary);
}
}
}
pub fn query_outcome(value: &Value) {
if let Some(qid) = value.get("qid").and_then(Value::as_str) {
println!("qid {qid}");
}
let local_total = value
.pointer("/local/total")
.and_then(Value::as_u64)
.unwrap_or(0);
println!("local: {local_total} match(es)");
if let Some(items) = value.pointer("/local/results").and_then(Value::as_array) {
for item in items {
print_item("local", item);
}
}
if let Some(responses) = value.get("responses").and_then(Value::as_array) {
println!("remote: {} response(s)", responses.len());
for response in responses {
let member = response
.get("member")
.and_then(Value::as_str)
.unwrap_or("unknown");
let short = member.get(..16).unwrap_or(member);
let truncated = response
.get("truncated")
.and_then(Value::as_bool)
.unwrap_or(false);
let more = response
.get("more_available")
.and_then(Value::as_u64)
.unwrap_or(0);
println!(" from member {short}... (truncated={truncated}, more_available={more})");
if let Some(results) = response.get("results").and_then(Value::as_array) {
for item in results {
print_item(short, item);
}
}
}
}
}
fn print_item(provenance: &str, item: &Value) {
let title = item.get("title").and_then(Value::as_str).unwrap_or("");
let url = item.get("url").and_then(Value::as_str).unwrap_or("");
let exposure = item.get("exposure").and_then(Value::as_str).unwrap_or("");
println!("- [{provenance}] {title} ({exposure})");
println!(" {url}");
}
+250
View File
@@ -0,0 +1,250 @@
mod common;
use std::fs;
use std::io::{BufRead, BufReader, Read};
use std::path::Path;
use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};
fn frxd() -> Command {
Command::new(env!("CARGO_BIN_EXE_frxd"))
}
fn frx() -> Command {
Command::new(env!("CARGO_BIN_EXE_frx"))
}
fn init_args(
config: &Path,
name: &str,
listen_port: u16,
relay_port: u16,
data: &Path,
) -> Vec<String> {
vec![
"--config".to_string(),
config.display().to_string(),
"init".to_string(),
"--name".to_string(),
name.to_string(),
"--listen".to_string(),
format!("127.0.0.1:{listen_port}"),
"--relay".to_string(),
format!("http://127.0.0.1:{relay_port}"),
"--data-dir".to_string(),
data.display().to_string(),
]
}
fn add_args(config: &Path, docs: &Path, name: &str, shared: bool) -> Vec<String> {
let mut args = vec![
"--config".to_string(),
config.display().to_string(),
"add".to_string(),
docs.display().to_string(),
"--name".to_string(),
name.to_string(),
"--exposure".to_string(),
"full".to_string(),
];
if shared {
args.push("--shared".to_string());
}
args
}
fn run_ok(args: &[String]) -> String {
let output = frxd().args(args).output().unwrap();
assert!(
output.status.success(),
"command failed: {}\n{}",
args.join(" "),
String::from_utf8_lossy(&output.stderr)
);
String::from_utf8_lossy(&output.stdout).to_string()
}
struct Service {
child: Child,
}
impl Drop for Service {
fn drop(&mut self) {
let _ = self.child.kill();
let _ = self.child.wait();
}
}
fn spawn_service(args: &[String], needle: &str) -> Service {
let mut child = frxd()
.args(args)
.stdout(Stdio::piped())
.stderr(Stdio::null())
.spawn()
.unwrap();
let reader = BufReader::new(child.stdout.take().unwrap());
let mut reader = reader;
let deadline = Instant::now() + Duration::from_secs(10);
loop {
let mut line = String::new();
let read = reader.read_line(&mut line).unwrap();
if read == 0 {
panic!("service exited before printing '{needle}'");
}
if line.contains(needle) {
break;
}
assert!(Instant::now() < deadline, "timeout waiting for '{needle}'");
}
std::thread::spawn(move || {
let mut sink = Vec::new();
let _ = reader.read_to_end(&mut sink);
});
Service { child }
}
#[test]
fn cli_init_add_search_status() {
let root = tempfile::tempdir().unwrap();
let config = root.path().join("frxd.toml");
let data = root.path().join("data");
let docs = root.path().join("docs");
fs::create_dir_all(&docs).unwrap();
fs::write(docs.join("note.txt"), "cli smoke rust document").unwrap();
let config_arg = config.display().to_string();
let stdout = run_ok(&init_args(&config, "alice", 0, 1, &data));
assert!(stdout.contains("pubkey"));
assert!(config.exists());
let key = data.join("key.hex");
assert!(key.exists());
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let mode = fs::metadata(&key).unwrap().permissions().mode() & 0o777;
assert_eq!(mode, 0o600, "key file must not be world readable");
}
let stdout = run_ok(&add_args(&config, &docs, "docs", true));
assert!(stdout.contains("indexed 1 file(s)"));
let stdout = run_ok(&[
"--config".to_string(),
config_arg.clone(),
"search".to_string(),
"rust".to_string(),
]);
assert!(stdout.contains("cli smoke rust document"));
let output = frx()
.args(["--config", &config_arg, "search", "rust"])
.output()
.unwrap();
assert!(output.status.success());
assert!(String::from_utf8_lossy(&output.stdout).contains("cli smoke rust document"));
let stdout = run_ok(&["--config".to_string(), config_arg, "status".to_string()]);
assert!(stdout.contains("node not running"));
}
#[test]
fn cli_init_refuses_overwrite_without_force() {
let root = tempfile::tempdir().unwrap();
let config = root.path().join("frxd.toml");
let data = root.path().join("data");
let args = init_args(&config, "alice", 0, 1, &data);
run_ok(&args);
let output = frxd().args(&args).output().unwrap();
assert!(!output.status.success());
assert!(String::from_utf8_lossy(&output.stderr).contains("already exists"));
let mut forced = args.clone();
forced.push("--force".to_string());
run_ok(&forced);
}
#[test]
fn cli_full_network_pipeline() {
let root = tempfile::tempdir().unwrap();
let relay_port = common::free_port();
let relay_args = vec![
"relay".to_string(),
"--listen".to_string(),
format!("127.0.0.1:{relay_port}"),
];
let _relay = spawn_service(&relay_args, "relay listening");
let bob_port = common::free_port();
let alice_port = common::free_port();
let bob_config = root.path().join("bob.toml");
let alice_config = root.path().join("alice.toml");
let bob_docs = root.path().join("bob-docs");
let alice_docs = root.path().join("alice-docs");
fs::create_dir_all(&bob_docs).unwrap();
fs::create_dir_all(&alice_docs).unwrap();
fs::write(
bob_docs.join("tantivy.md"),
"# Tantivy BM25\nTantivy indexes rust documents.",
)
.unwrap();
fs::write(alice_docs.join("private.txt"), "alice private rust note").unwrap();
run_ok(&init_args(
&bob_config,
"bob",
bob_port,
relay_port,
&root.path().join("bob-data"),
));
run_ok(&init_args(
&alice_config,
"alice",
alice_port,
relay_port,
&root.path().join("alice-data"),
));
run_ok(&add_args(&bob_config, &bob_docs, "shared", true));
run_ok(&add_args(&alice_config, &alice_docs, "mine", false));
let bob_serve = vec![
"--config".to_string(),
bob_config.display().to_string(),
"serve".to_string(),
];
let alice_serve = vec![
"--config".to_string(),
alice_config.display().to_string(),
"serve".to_string(),
];
let _bob = spawn_service(&bob_serve, "frxd listening");
let _alice = spawn_service(&alice_serve, "frxd listening");
std::thread::sleep(Duration::from_millis(400));
let alice_config_arg = alice_config.display().to_string();
let stdout = run_ok(&[
"--config".to_string(),
alice_config_arg.clone(),
"query".to_string(),
"tantivy".to_string(),
]);
assert!(stdout.contains("Tantivy BM25"), "{stdout}");
assert!(stdout.contains("remote: 1 response(s)"), "{stdout}");
let stdout = run_ok(&[
"--config".to_string(),
alice_config_arg.clone(),
"query".to_string(),
"rust".to_string(),
"--local-only".to_string(),
]);
assert!(stdout.contains("alice private rust note"), "{stdout}");
assert!(stdout.contains("remote: 0 response(s)"), "{stdout}");
let output = frx()
.args(["--config", &alice_config_arg, "query", "tantivy"])
.output()
.unwrap();
assert!(output.status.success());
assert!(String::from_utf8_lossy(&output.stdout).contains("Tantivy BM25"));
}
+163
View File
@@ -0,0 +1,163 @@
#![allow(dead_code)]
use std::path::Path;
use std::time::Duration;
use frxd::config::{Config, IndexSection, NodeSection, QuerySection};
use frxd::crypto::Keypair;
use frxd::index::Collection;
use frxd::message::{Envelope, QueryBody, TYPE_QUERY};
use frxd::relay;
use serde_json::Value;
pub fn config_for(dir: &Path, name: &str, relay_url: &str) -> Config {
let config = Config {
node: NodeSection {
name: name.to_string(),
listen: "127.0.0.1:0".to_string(),
relays: vec![relay_url.to_string()],
trusted_keys: Vec::new(),
responder: true,
},
query: QuerySection {
max_results: 5,
timeout_ms: 700,
},
index: IndexSection {
data_dir: dir.join("data").display().to_string(),
},
};
config.save_key(&Keypair::generate()).unwrap();
config
}
pub fn collection(name: &str, path: &Path, shared: bool, exposure: &str) -> Collection {
Collection {
name: name.to_string(),
path: path.display().to_string(),
shared,
exposure: exposure.to_string(),
}
}
pub async fn spawn_relay() -> String {
spawn_relay_with_capacity(256).await
}
pub async fn spawn_relay_with_capacity(capacity: usize) -> String {
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
tokio::spawn(async move {
let _ = relay::run(listener, capacity).await;
});
format!("http://{addr}")
}
pub fn free_port() -> u16 {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
listener.local_addr().unwrap().port()
}
pub fn query_envelope(key: &Keypair, text: &str, max_results: usize) -> Envelope {
let body = QueryBody::new(text, max_results);
Envelope::new(key, TYPE_QUERY, serde_json::to_value(&body).unwrap())
}
pub async fn register(client: &reqwest::Client, relay_url: &str, member: &str) {
poll(client, relay_url, member, 30).await;
}
pub async fn poll_messages(
client: &reqwest::Client,
relay_url: &str,
member: &str,
timeout_ms: u64,
) -> Vec<Value> {
let response = poll(client, relay_url, member, timeout_ms).await;
if response.status() == reqwest::StatusCode::NO_CONTENT {
return Vec::new();
}
let payload: Value = response.json().await.unwrap();
payload
.get("messages")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default()
}
pub fn messages_of_type(messages: &[Value], msg_type: &str) -> Vec<Value> {
messages
.iter()
.filter(|m| m.get("type").and_then(Value::as_str) == Some(msg_type))
.cloned()
.collect()
}
pub fn client() -> reqwest::Client {
reqwest::Client::builder()
.timeout(Duration::from_secs(10))
.build()
.unwrap()
}
pub async fn ask(
client: &reqwest::Client,
addr: &str,
text: &str,
max: usize,
) -> (reqwest::StatusCode, String, Value) {
let response = client
.post(format!("http://{addr}/v1/local/query"))
.json(&serde_json::json!({
"text": text,
"max_results": max,
"timeout_ms": 700,
}))
.send()
.await
.unwrap();
let status = response.status();
let raw = response.text().await.unwrap();
let value: Value = serde_json::from_str(&raw).unwrap();
(status, raw, value)
}
pub async fn publish(
client: &reqwest::Client,
relay_url: &str,
envelope: &Envelope,
) -> reqwest::Response {
client
.post(format!("{relay_url}/v1/publish"))
.json(envelope)
.send()
.await
.unwrap()
}
pub async fn unicast(
client: &reqwest::Client,
relay_url: &str,
to: &str,
envelope: &Envelope,
) -> reqwest::Response {
client
.post(format!("{relay_url}/v1/unicast?to={to}"))
.json(envelope)
.send()
.await
.unwrap()
}
pub async fn poll(
client: &reqwest::Client,
relay_url: &str,
member: &str,
timeout_ms: u64,
) -> reqwest::Response {
client
.get(format!(
"{relay_url}/v1/poll?member={member}&timeout_ms={timeout_ms}"
))
.send()
.await
.unwrap()
}
+210
View File
@@ -0,0 +1,210 @@
mod common;
use std::collections::BTreeSet;
use std::fs;
use std::path::Path;
use std::time::{Duration, Instant};
use common::{
ask, client, collection, config_for, poll_messages, publish, query_envelope, register,
spawn_relay,
};
use frxd::crypto::Keypair;
use frxd::index::LocalIndex;
use frxd::message::{EXPOSURE_FULL, TYPE_RESPONSE};
use frxd::node::{self, Node, NodeHandle};
use serde_json::Value;
fn corpus_dir(root: &Path, name: &str, files: &[(&str, &str)]) -> std::path::PathBuf {
let dir = root.join(format!("{name}-docs"));
fs::create_dir_all(&dir).unwrap();
for (file, text) in files {
fs::write(dir.join(file), text).unwrap();
}
dir
}
async fn start_node_with_corpus(
root: &Path,
name: &str,
relays: Vec<String>,
files: &[(&str, &str)],
) -> NodeHandle {
let docs = corpus_dir(root, name, files);
let mut config = config_for(&root.join(name), name, relays.first().unwrap());
config.node.relays = relays;
{
let index = LocalIndex::open(&config.index_dir()).unwrap();
index
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
.unwrap();
}
Node::start(config).await.unwrap()
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_queries_each_get_their_response() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
vec![relay_url.clone()],
&[("doc.txt", "concurrent rust document")],
)
.await;
let alice = start_node_with_corpus(
root.path(),
"alice",
vec![relay_url.clone()],
&[("mine.txt", "alice notes")],
)
.await;
let http = client();
let addr = alice.addr.to_string();
let mut tasks = Vec::new();
for _ in 0..4 {
let http = http.clone();
let addr = addr.clone();
tasks.push(tokio::spawn(
async move { ask(&http, &addr, "rust", 5).await },
));
}
let mut qids = BTreeSet::new();
for task in tasks {
let (_status, _raw, value) = task.await.unwrap();
let qid = value
.get("qid")
.and_then(Value::as_str)
.unwrap()
.to_string();
qids.insert(qid);
let responses = value.get("responses").and_then(Value::as_array).unwrap();
assert_eq!(responses.len(), 1, "query lost its response: {value}");
}
assert_eq!(qids.len(), 4);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn multi_relay_deduplicates_and_responds_once() {
let root = tempfile::tempdir().unwrap();
let relay_one = spawn_relay().await;
let relay_two = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
vec![relay_one.clone(), relay_two.clone()],
&[("doc.txt", "dual relay rust document")],
)
.await;
let http = client();
let alice = Keypair::generate();
register(&http, &relay_one, &alice.public_hex()).await;
register(&http, &relay_two, &alice.public_hex()).await;
let envelope = query_envelope(&alice, "rust", 5);
assert!(
publish(&http, &relay_one, &envelope)
.await
.status()
.is_success()
);
assert!(
publish(&http, &relay_two, &envelope)
.await
.status()
.is_success()
);
let deadline = Instant::now() + Duration::from_millis(1200);
let mut responses = Vec::new();
while responses.is_empty() && Instant::now() < deadline {
for relay in [&relay_one, &relay_two] {
let messages = poll_messages(&http, relay, &alice.public_hex(), 200).await;
responses.extend(
messages
.iter()
.filter(|m| m.get("type").and_then(Value::as_str) == Some(TYPE_RESPONSE))
.cloned(),
);
}
}
assert_eq!(
responses.len(),
1,
"duplicate relay delivery was answered twice"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn index_persists_across_node_restart() {
let root = tempfile::tempdir().unwrap();
let docs = corpus_dir(
root.path(),
"bob",
&[("doc.txt", "persistent rust document")],
);
let config = config_for(&root.path().join("bob"), "bob", "http://127.0.0.1:1");
{
let index = LocalIndex::open(&config.index_dir()).unwrap();
index
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
.unwrap();
}
let first = Node::start(config.clone()).await.unwrap();
let outcome = node::control_query(
&format!("http://{}", first.addr),
"rust",
Some(5),
Some(100),
false,
)
.await
.unwrap();
assert_eq!(
outcome.pointer("/local/total").and_then(Value::as_u64),
Some(1)
);
drop(first);
let second = Node::start(config).await.unwrap();
let outcome = node::control_query(
&format!("http://{}", second.addr),
"rust",
Some(5),
Some(100),
false,
)
.await
.unwrap();
assert_eq!(
outcome.pointer("/local/total").and_then(Value::as_u64),
Some(1)
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn poll_returns_queued_batch_in_one_call() {
let relay_url = spawn_relay().await;
let http = client();
let member = Keypair::generate();
register(&http, &relay_url, &member.public_hex()).await;
for index in 0..3 {
let envelope = query_envelope(&member, &format!("query {index}"), 5);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
}
let messages = poll_messages(&http, &relay_url, &member.public_hex(), 300).await;
assert_eq!(messages.len(), 3);
assert!(
poll_messages(&http, &relay_url, &member.public_hex(), 50)
.await
.is_empty()
);
}
+924
View File
@@ -0,0 +1,924 @@
mod common;
use std::collections::BTreeSet;
use std::fs;
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use common::{
ask, client, collection, config_for, messages_of_type, poll, poll_messages, publish,
query_envelope, register, spawn_relay, spawn_relay_with_capacity, unicast,
};
use frxd::crypto::Keypair;
use frxd::index::LocalIndex;
use frxd::message::{
EXPOSURE_FULL, EXPOSURE_METADATA, Envelope, QueryBody, TYPE_AGGREGATE, TYPE_QUERY,
TYPE_RESPONSE,
};
use frxd::node::{Node, NodeHandle};
use serde_json::{Value, json};
fn keys(value: &Value) -> BTreeSet<String> {
value.as_object().expect("object").keys().cloned().collect()
}
async fn status(http: &reqwest::Client, addr: &str) -> Value {
http.get(format!("http://{addr}/v1/local/status"))
.send()
.await
.unwrap()
.json()
.await
.unwrap()
}
fn corpus_dir(root: &Path, name: &str, files: &[(&str, &str)]) -> PathBuf {
let dir = root.join(format!("{name}-docs"));
fs::create_dir_all(&dir).unwrap();
for (file, text) in files {
fs::write(dir.join(file), text).unwrap();
}
dir
}
async fn start_node_with_corpus(
root: &Path,
name: &str,
relay_url: &str,
shared: bool,
exposure: &str,
files: &[(&str, &str)],
) -> NodeHandle {
let docs = corpus_dir(root, name, files);
let config = config_for(&root.join(name), name, relay_url);
{
let index = LocalIndex::open(&config.index_dir()).unwrap();
index
.add_collection(&collection("docs", &docs, shared, exposure))
.unwrap();
}
Node::start(config).await.unwrap()
}
async fn collect_responses(
http: &reqwest::Client,
relay_url: &str,
member: &str,
timeout_ms: u64,
) -> Vec<Value> {
let start = Instant::now();
let deadline = Duration::from_millis(timeout_ms);
let mut responses = Vec::new();
while start.elapsed() < deadline {
let messages = poll_messages(http, relay_url, member, 200).await;
responses.extend(messages_of_type(&messages, TYPE_RESPONSE));
if !responses.is_empty() {
break;
}
}
responses
}
async fn raw_query(
http: &reqwest::Client,
relay_url: &str,
asker: &Keypair,
text: &str,
timeout_ms: u64,
) -> Vec<Value> {
register(http, relay_url, &asker.public_hex()).await;
let envelope = query_envelope(asker, text, 5);
assert!(
publish(http, relay_url, &envelope)
.await
.status()
.is_success()
);
collect_responses(http, relay_url, &asker.public_hex(), timeout_ms).await
}
#[test]
fn envelope_field_set_is_fixed() {
let key = Keypair::generate();
let envelope = Envelope::new(&key, TYPE_QUERY, json!({"qid": "q", "text": "t"}));
let value = serde_json::to_value(&envelope).unwrap();
assert_eq!(
keys(&value),
BTreeSet::from([
"body".to_string(),
"from".to_string(),
"nonce".to_string(),
"sig".to_string(),
"ts".to_string(),
"type".to_string(),
])
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn ordering_travels_scores_dont() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[
("strong.txt", "rust rust rust rust search engine"),
("weak.txt", "rust notes"),
],
)
.await;
let alice = Keypair::generate();
let http = client();
let responses = raw_query(&http, &relay_url, &alice, "rust", 700).await;
assert_eq!(responses.len(), 1);
let results = responses[0]
.pointer("/body/results")
.and_then(Value::as_array)
.unwrap();
assert_eq!(results.len(), 2);
let first_url = results[0].get("url").and_then(Value::as_str).unwrap();
assert!(
first_url.contains("strong"),
"best match should lead the ordering: {first_url}"
);
for result in results {
assert!(!keys(result).contains("score"));
assert!(!keys(result).contains("rank"));
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn metadata_exposure_withholds_content() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_METADATA,
&[("corpus.txt", "enrichment corpus about rust")],
)
.await;
let alice = Keypair::generate();
let http = client();
let responses = raw_query(&http, &relay_url, &alice, "rust", 700).await;
assert_eq!(responses.len(), 1);
let result = &responses[0]
.pointer("/body/results")
.and_then(Value::as_array)
.unwrap()[0];
assert_eq!(
result.get("exposure").and_then(Value::as_str),
Some(EXPOSURE_METADATA)
);
assert!(result.get("content").unwrap().is_null());
assert!(
!result
.get("summary")
.and_then(Value::as_str)
.unwrap()
.is_empty()
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn full_exposure_carries_content() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("corpus.txt", "full exposure rust content body")],
)
.await;
let alice = Keypair::generate();
let http = client();
let responses = raw_query(&http, &relay_url, &alice, "rust", 700).await;
let result = &responses[0]
.pointer("/body/results")
.and_then(Value::as_array)
.unwrap()[0];
assert_eq!(
result.get("exposure").and_then(Value::as_str),
Some(EXPOSURE_FULL)
);
assert!(
result
.get("content")
.and_then(Value::as_str)
.unwrap()
.contains("full exposure rust content body")
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn private_collections_are_silent_egress() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
false,
EXPOSURE_FULL,
&[("private.txt", "rust private data")],
)
.await;
let alice = Keypair::generate();
let http = client();
let responses = raw_query(&http, &relay_url, &alice, "rust", 500).await;
assert!(responses.is_empty(), "private collection answered");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn one_result_is_a_conformant_response() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("only.txt", "solitary rust document")],
)
.await;
let alice = Keypair::generate();
let http = client();
let responses = raw_query(&http, &relay_url, &alice, "solitary", 700).await;
assert_eq!(responses.len(), 1);
assert_eq!(
responses[0]
.pointer("/body/results")
.and_then(Value::as_array)
.unwrap()
.len(),
1
);
assert_eq!(
responses[0]
.pointer("/body/truncated")
.and_then(Value::as_bool),
Some(false)
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn empty_query_earns_silence() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "rust content")],
)
.await;
let alice = start_node_with_corpus(
root.path(),
"alice",
&relay_url,
false,
EXPOSURE_FULL,
&[("mine.txt", "alice rust")],
)
.await;
let response = client()
.post(format!("http://{}/v1/local/query", alice.addr))
.json(&json!({"text": "", "max_results": 5, "timeout_ms": 500}))
.send()
.await
.unwrap()
.json::<Value>()
.await
.unwrap();
assert!(
response
.get("responses")
.and_then(Value::as_array)
.unwrap()
.is_empty()
);
assert!(bob.node.doc_count() > 0);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn entities_are_optional_hints() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "entity tagged rust document")],
)
.await;
let alice = Keypair::generate();
let http = client();
register(&http, &relay_url, &alice.public_hex()).await;
let body = json!({
"qid": "entities-test",
"text": "rust",
"entities": ["Q999999", "not-a-real-entity"],
"budget": {"max_results": 5}
});
let envelope = Envelope::new(&alice, TYPE_QUERY, body);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let responses = collect_responses(&http, &relay_url, &alice.public_hex(), 700).await;
assert_eq!(responses.len(), 1);
assert_eq!(
responses[0].pointer("/body/qid").and_then(Value::as_str),
Some("entities-test")
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn trusted_keys_allowlist_filters_senders() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let alice = Keypair::generate();
let untrusted = Keypair::generate();
let docs = corpus_dir(root.path(), "bob", &[("doc.txt", "rust document")]);
let mut config = config_for(&root.path().join("bob"), "bob", &relay_url);
config.node.trusted_keys = vec![alice.public_hex()];
{
let index = LocalIndex::open(&config.index_dir()).unwrap();
index
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
.unwrap();
}
let _bob = Node::start(config).await.unwrap();
let http = client();
let trusted = raw_query(&http, &relay_url, &alice, "rust", 700).await;
assert_eq!(trusted.len(), 1, "trusted member got no response");
register(&http, &relay_url, &untrusted.public_hex()).await;
let envelope = query_envelope(&untrusted, "rust", 5);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let denied = collect_responses(&http, &relay_url, &untrusted.public_hex(), 400).await;
assert!(denied.is_empty(), "untrusted member received a response");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn backpressure_is_visible_and_recoverable() {
let relay_url = spawn_relay_with_capacity(1).await;
let http = client();
let member = Keypair::generate();
let publisher = Keypair::generate();
register(&http, &relay_url, &member.public_hex()).await;
let first = query_envelope(&publisher, "one", 5);
assert_eq!(
publish(&http, &relay_url, &first).await.status(),
reqwest::StatusCode::ACCEPTED
);
let second = query_envelope(&publisher, "two", 5);
let response = publish(&http, &relay_url, &second).await;
assert_eq!(response.status(), reqwest::StatusCode::TOO_MANY_REQUESTS);
assert_eq!(
response
.headers()
.get("retry-after")
.and_then(|v| v.to_str().ok()),
Some("1")
);
assert_eq!(
poll_messages(&http, &relay_url, &member.public_hex(), 300)
.await
.len(),
1
);
assert_eq!(
poll_messages(&http, &relay_url, &publisher.public_hex(), 300)
.await
.len(),
1
);
let third = query_envelope(&publisher, "three", 5);
assert_eq!(
publish(&http, &relay_url, &third).await.status(),
reqwest::StatusCode::ACCEPTED
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn unicast_is_need_to_know() {
let relay_url = spawn_relay().await;
let http = client();
let alice = Keypair::generate();
let bob = Keypair::generate();
register(&http, &relay_url, &alice.public_hex()).await;
register(&http, &relay_url, &bob.public_hex()).await;
let query = query_envelope(&alice, "rust", 5);
assert!(
publish(&http, &relay_url, &query)
.await
.status()
.is_success()
);
let response = Envelope::new(
&bob,
TYPE_RESPONSE,
json!({"qid": "q1", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
);
assert_eq!(
unicast(&http, &relay_url, &alice.public_hex(), &response)
.await
.status(),
reqwest::StatusCode::OK
);
let alice_messages = poll_messages(&http, &relay_url, &alice.public_hex(), 300).await;
assert_eq!(messages_of_type(&alice_messages, TYPE_RESPONSE).len(), 1);
let bob_messages = poll_messages(&http, &relay_url, &bob.public_hex(), 300).await;
assert!(
messages_of_type(&bob_messages, TYPE_RESPONSE).is_empty(),
"unicast response leaked to another member"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn unicast_to_unknown_member_is_visible() {
let relay_url = spawn_relay().await;
let http = client();
let sender = Keypair::generate();
let response = Envelope::new(
&sender,
TYPE_RESPONSE,
json!({"qid": "q1", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
);
let stranger = Keypair::generate().public_hex();
assert_eq!(
unicast(&http, &relay_url, &stranger, &response)
.await
.status(),
reqwest::StatusCode::NOT_FOUND
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn tampered_envelopes_are_rejected() {
let relay_url = spawn_relay().await;
let http = client();
let key = Keypair::generate();
let mut envelope = query_envelope(&key, "rust", 5);
envelope.body = json!({"qid": "q1", "text": "forged", "budget": {"max_results": 5}});
let publish_response = publish(&http, &relay_url, &envelope).await;
assert_eq!(publish_response.status(), reqwest::StatusCode::BAD_REQUEST);
let mut response = Envelope::new(
&key,
TYPE_RESPONSE,
json!({"qid": "q1", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
);
response.sig = "00".repeat(64);
let unicast_response = unicast(&http, &relay_url, &key.public_hex(), &response).await;
assert_eq!(unicast_response.status(), reqwest::StatusCode::BAD_REQUEST);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn invalid_poll_member_is_rejected() {
let relay_url = spawn_relay().await;
let response = poll(&client(), &relay_url, "not-hex-at-all", 30).await;
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn malformed_json_is_rejected() {
let relay_url = spawn_relay().await;
let http = client();
let response = http
.post(format!("{relay_url}/v1/publish"))
.header("content-type", "application/json")
.body("{not json")
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
let root = tempfile::tempdir().unwrap();
let config = config_for(&root.path().join("alice"), "alice", &relay_url);
let node = Node::start(config).await.unwrap();
let response = http
.post(format!("http://{}/v1/local/query", node.addr))
.header("content-type", "application/json")
.body("{not json")
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn node_does_not_answer_its_own_queries() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "self query rust")],
)
.await;
let (_status, _raw, value) = ask(&client(), &bob.addr.to_string(), "rust", 5).await;
assert!(
value
.get("responses")
.and_then(Value::as_array)
.unwrap()
.is_empty(),
"node responded to itself: {value}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn ingress_happens_only_locally() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "rust served by bob")],
)
.await;
let alice = start_node_with_corpus(
root.path(),
"alice",
&relay_url,
false,
EXPOSURE_FULL,
&[("mine.txt", "alice rust notes")],
)
.await;
let http = client();
let bob_before = status(&http, &bob.addr.to_string()).await;
let alice_before = status(&http, &alice.addr.to_string()).await;
let (_s, _r, value) = ask(&http, &alice.addr.to_string(), "rust", 5).await;
assert_eq!(
value
.get("responses")
.and_then(Value::as_array)
.unwrap()
.len(),
1
);
let bob_after = status(&http, &bob.addr.to_string()).await;
let alice_after = status(&http, &alice.addr.to_string()).await;
assert_eq!(bob_before.get("doc_count"), bob_after.get("doc_count"));
assert_eq!(alice_before.get("doc_count"), alice_after.get("doc_count"));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn querier_and_responder_roles_coexist() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let alice = start_node_with_corpus(
root.path(),
"alice",
&relay_url,
true,
EXPOSURE_FULL,
&[("a.txt", "alice rust document")],
)
.await;
let bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("b.txt", "bob rust document")],
)
.await;
let http = client();
let alice_addr = alice.addr.to_string();
let bob_addr = bob.addr.to_string();
let alice_ask = async { common::ask(&http, &alice_addr, "rust", 5).await };
let bob_ask = async { common::ask(&http, &bob_addr, "rust", 5).await };
let (alice_result, bob_result) = tokio::join!(alice_ask, bob_ask);
for (name, (_status, _raw, value)) in [("alice", alice_result), ("bob", bob_result)] {
let responses = value.get("responses").and_then(Value::as_array).unwrap();
assert_eq!(responses.len(), 1, "{name} got {value}");
let member = responses[0].get("member").and_then(Value::as_str).unwrap();
assert!(!member.is_empty());
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn budget_is_clamped_and_never_exceeded() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[
("one.txt", "rust one"),
("two.txt", "rust two"),
("three.txt", "rust three"),
],
)
.await;
let alice = start_node_with_corpus(
root.path(),
"alice",
&relay_url,
false,
EXPOSURE_FULL,
&[("mine.txt", "alice doc")],
)
.await;
let http = client();
let (_s, _r, value) = ask(&http, &alice.addr.to_string(), "rust", 0).await;
let responses = value.get("responses").and_then(Value::as_array).unwrap();
assert_eq!(responses.len(), 1);
assert_eq!(
responses[0]
.get("results")
.and_then(Value::as_array)
.unwrap()
.len(),
1,
"zero budget must clamp to one, never send nothing or more"
);
let (_s, _r, value) = ask(&http, &alice.addr.to_string(), "rust", 1000).await;
let responses = value.get("responses").and_then(Value::as_array).unwrap();
assert_eq!(
responses[0]
.get("results")
.and_then(Value::as_array)
.unwrap()
.len(),
3
);
assert_eq!(
responses[0].get("truncated").and_then(Value::as_bool),
Some(false)
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn aggregates_are_unicast_only_and_ignored_by_nodes() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "aggregate test rust")],
)
.await;
let http = client();
let sender = Keypair::generate();
register(&http, &relay_url, &bob.pubkey).await;
let broadcast = Envelope::new(
&sender,
TYPE_AGGREGATE,
json!({"period": "2026-03", "sent": 1, "passed": 0, "cited": 0}),
);
assert_eq!(
publish(&http, &relay_url, &broadcast).await.status(),
reqwest::StatusCode::BAD_REQUEST,
"aggregates must not travel the broadcast channel"
);
let unicast_envelope = Envelope::new(
&sender,
TYPE_AGGREGATE,
json!({"period": "2026-03", "sent": 1, "passed": 0, "cited": 0}),
);
assert_eq!(
unicast(&http, &relay_url, &bob.pubkey, &unicast_envelope)
.await
.status(),
reqwest::StatusCode::OK
);
tokio::time::sleep(Duration::from_millis(200)).await;
let (_s, _r, value) = ask(&http, &bob.addr.to_string(), "ignored", 5).await;
assert_eq!(
value.pointer("/local/total").and_then(Value::as_u64),
Some(0)
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn node_survives_unreachable_relay() {
let root = tempfile::tempdir().unwrap();
let alice = start_node_with_corpus(
root.path(),
"alice",
"http://127.0.0.1:1",
false,
EXPOSURE_FULL,
&[("mine.txt", "offline rust note")],
)
.await;
let (status_code, _raw, value) = ask(&client(), &alice.addr.to_string(), "rust", 5).await;
assert!(status_code.is_success());
assert_eq!(
value.pointer("/local/total").and_then(Value::as_u64),
Some(1)
);
assert!(
value
.get("responses")
.and_then(Value::as_array)
.unwrap()
.is_empty()
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn duplicate_delivery_is_answered_once() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "duplicate rust doc")],
)
.await;
let alice = Keypair::generate();
let http = client();
register(&http, &relay_url, &alice.public_hex()).await;
let envelope = query_envelope(&alice, "rust", 5);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let responses = collect_responses(&http, &relay_url, &alice.public_hex(), 700).await;
assert_eq!(
responses.len(),
1,
"duplicate query produced duplicate responses"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn multiple_responders_merge_with_provenance() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("bob.txt", "bob rust contribution")],
)
.await;
let _carol = start_node_with_corpus(
root.path(),
"carol",
&relay_url,
true,
EXPOSURE_FULL,
&[("carol.txt", "carol rust contribution")],
)
.await;
let alice = start_node_with_corpus(
root.path(),
"alice",
&relay_url,
false,
EXPOSURE_FULL,
&[("mine.txt", "alice rust local")],
)
.await;
let (_s, _r, value) = ask(&client(), &alice.addr.to_string(), "rust", 5).await;
let responses = value.get("responses").and_then(Value::as_array).unwrap();
assert_eq!(responses.len(), 2, "expected both responders: {value}");
let members: BTreeSet<&str> = responses
.iter()
.map(|r| r.get("member").and_then(Value::as_str).unwrap())
.collect();
assert_eq!(members.len(), 2);
let merged = value.get("merged").and_then(Value::as_array).unwrap();
let provenances: BTreeSet<&str> = merged
.iter()
.map(|m| m.get("provenance").and_then(Value::as_str).unwrap())
.collect();
assert!(provenances.contains("local"));
assert_eq!(provenances.len(), 3);
}
#[test]
fn canonical_signing_bytes_are_stable() {
let mut envelope = Envelope {
msg_type: TYPE_QUERY.to_string(),
from: "aa".repeat(32),
ts: 1_700_000_000,
nonce: "00112233445566778899aabbccddeeff".to_string(),
body: json!({"text": "rust", "qid": "q1", "budget": {"max_results": 5}, "entities": []}),
sig: String::new(),
};
let first = frxd::crypto::signing_bytes(&envelope);
envelope.body =
json!({"entities": [], "budget": {"max_results": 5}, "qid": "q1", "text": "rust"});
let second = frxd::crypto::signing_bytes(&envelope);
assert_eq!(first, second);
assert_eq!(
String::from_utf8(first).unwrap(),
"FRX/0.3\nquery\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n1700000000\n00112233445566778899aabbccddeeff\n{\"budget\":{\"max_results\":5},\"entities\":[],\"qid\":\"q1\",\"text\":\"rust\"}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn query_body_without_entities_parses() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let _bob = start_node_with_corpus(
root.path(),
"bob",
&relay_url,
true,
EXPOSURE_FULL,
&[("doc.txt", "minimal body rust")],
)
.await;
let alice = Keypair::generate();
let http = client();
register(&http, &relay_url, &alice.public_hex()).await;
let envelope = Envelope::new(
&alice,
TYPE_QUERY,
json!({"qid": "min", "text": "rust", "budget": {"max_results": 5}}),
);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let responses = collect_responses(&http, &relay_url, &alice.public_hex(), 700).await;
assert_eq!(responses.len(), 1);
}
#[test]
fn query_body_budget_is_the_only_quantity_knob() {
let value = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
assert_eq!(
keys(&value),
BTreeSet::from([
"qid".to_string(),
"text".to_string(),
"entities".to_string(),
"budget".to_string()
])
);
}
+116
View File
@@ -0,0 +1,116 @@
mod common;
use std::fs;
use common::{ask, client, collection, config_for, publish, spawn_relay};
use frxd::index::LocalIndex;
use frxd::message::{EXPOSURE_FULL, Envelope, TYPE_RESPONSE};
use frxd::node::Node;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn two_node_broadcast_query_flow() {
let root = tempfile::tempdir().unwrap();
let relay_url = spawn_relay().await;
let shared_dir = root.path().join("bob/shared");
let private_dir = root.path().join("bob/private");
fs::create_dir_all(&shared_dir).unwrap();
fs::create_dir_all(&private_dir).unwrap();
fs::write(
shared_dir.join("alpha.txt"),
"Tantivy provides BM25 relevance scoring for rust search",
)
.unwrap();
fs::write(
shared_dir.join("beta.txt"),
"Rust lifetimes and borrowing explained",
)
.unwrap();
fs::write(private_dir.join("gamma.txt"), "rust secret launch codes").unwrap();
let bob_config = config_for(&root.path().join("bob"), "bob", &relay_url);
{
let index = LocalIndex::open(&bob_config.index_dir()).unwrap();
index
.add_collection(&collection("shared", &shared_dir, true, EXPOSURE_FULL))
.unwrap();
index
.add_collection(&collection("private", &private_dir, false, "metadata"))
.unwrap();
}
let _bob = Node::start(bob_config).await.unwrap();
let alice_dir = root.path().join("alice");
let alice_local = root.path().join("alice_mine");
fs::create_dir_all(&alice_local).unwrap();
fs::write(alice_local.join("notes.txt"), "my rust notes").unwrap();
let alice_config = config_for(&alice_dir, "alice", &relay_url);
{
let index = LocalIndex::open(&alice_config.index_dir()).unwrap();
index
.add_collection(&collection("mine", &alice_local, false, "metadata"))
.unwrap();
}
let alice = Node::start(alice_config).await.unwrap();
let http = client();
let addr = alice.addr.to_string();
let (status, raw, value) = ask(&http, &addr, "rust", 1).await;
assert!(status.is_success());
assert_eq!(
value.pointer("/local/total").and_then(|v| v.as_u64()),
Some(1)
);
let responses = value.get("responses").and_then(|v| v.as_array()).unwrap();
assert_eq!(responses.len(), 1, "expected one responder: {value}");
let response = &responses[0];
let results = response.get("results").and_then(|v| v.as_array()).unwrap();
assert_eq!(results.len(), 1);
assert_eq!(
response.get("truncated").and_then(|v| v.as_bool()),
Some(true)
);
assert_eq!(
response.get("more_available").and_then(|v| v.as_u64()),
Some(1)
);
assert!(!raw.contains("gamma"), "unshared collection leaked: {raw}");
assert!(!raw.contains("\"score\""), "response carries scores: {raw}");
let (_status, _raw, value) = ask(&http, &addr, "rust", 5).await;
let responses = value.get("responses").and_then(|v| v.as_array()).unwrap();
assert_eq!(responses.len(), 1);
assert_eq!(
responses[0]
.get("results")
.and_then(|v| v.as_array())
.unwrap()
.len(),
2
);
assert_eq!(
responses[0].get("truncated").and_then(|v| v.as_bool()),
Some(false)
);
let (_status, _raw, value) = ask(&http, &addr, "secret launch codes", 5).await;
let responses = value.get("responses").and_then(|v| v.as_array()).unwrap();
assert!(
responses.is_empty(),
"private collection was served: {value}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn relay_rejects_non_query_broadcasts() {
let relay_url = spawn_relay().await;
let key = frxd::crypto::Keypair::generate();
let envelope = Envelope::new(
&key,
TYPE_RESPONSE,
serde_json::json!({"qid": "x", "results": []}),
);
let response = publish(&client(), &relay_url, &envelope).await;
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
}
+389
View File
@@ -0,0 +1,389 @@
mod common;
use std::collections::BTreeSet;
use std::fs;
use common::{client, collection, config_for, poll, publish, spawn_relay, unicast};
use frxd::crypto::Keypair;
use frxd::index::LocalIndex;
use frxd::message::{
AggregateBody, Envelope, QueryBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE,
build_response,
};
use frxd::node::{self, Node};
use serde_json::Value;
fn assert_exact_keys(value: &Value, expected: &[&str]) {
let actual: BTreeSet<&str> = value
.as_object()
.expect("object")
.keys()
.map(String::as_str)
.collect();
let expected: BTreeSet<&str> = expected.iter().copied().collect();
assert_eq!(actual, expected, "fields drifted: {value}");
}
fn assert_absent_fields(value: &Value, banned: &[&str]) {
let raw = serde_json::to_string(value).unwrap();
for field in banned {
assert!(
!raw.contains(&format!("\"{field}\"")),
"purged field '{field}' present: {raw}"
);
}
}
#[test]
fn no_bounty_winner_selection_or_slashing() {
let query = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
assert_exact_keys(&query, &["qid", "text", "entities", "budget"]);
assert_exact_keys(query.get("budget").unwrap(), &["max_results"]);
let envelope = Envelope::new(&Keypair::generate(), TYPE_QUERY, query.clone());
let envelope = serde_json::to_value(&envelope).unwrap();
for value in [&query, &envelope] {
assert_absent_fields(
value,
&[
"bounty",
"winner",
"winner_selection",
"slashing",
"stake",
"reward",
"payment",
],
);
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_protocol_query_dedup() {
let relay_url = spawn_relay().await;
let http = client();
let key = Keypair::generate();
let envelope = Envelope::new(
&key,
TYPE_QUERY,
serde_json::json!({"qid": "q1", "text": "rust"}),
);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let response = poll(&http, &relay_url, &key.public_hex(), 300).await;
let payload: Value = response.json().await.unwrap();
assert_eq!(
payload
.get("messages")
.and_then(Value::as_array)
.map(Vec::len),
Some(2),
"relay deduplicated identical queries: {payload}"
);
}
#[test]
fn no_k_fetch_ingestion_attestations() {
let response = build_response("q1", Vec::new(), 0, 5);
let value = serde_json::to_value(&response).unwrap();
assert_exact_keys(
&value,
&["qid", "results", "truncated", "more_available", "cursor"],
);
assert_absent_fields(
&value,
&[
"attestation",
"attestations",
"k_fetch",
"fetch_proof",
"ingestion_proof",
"fetch_count",
],
);
}
#[test]
fn no_result_count_etiquette() {
let query = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
assert_absent_fields(
&query,
&["min_results", "results_count", "serp", "count_floor"],
);
let response = serde_json::to_value(build_response("q1", Vec::new(), 0, 5)).unwrap();
assert_absent_fields(
&response,
&["min_results", "results_count", "serp", "count_floor"],
);
}
#[test]
fn no_global_reputation_score() {
let item = ResponseItem {
url: "file:///x".to_string(),
title: "t".to_string(),
summary: String::new(),
published: String::new(),
exposure: "metadata".to_string(),
content: None,
};
let item_value = serde_json::to_value(&item).unwrap();
assert_exact_keys(
&item_value,
&[
"url",
"title",
"summary",
"published",
"exposure",
"content",
],
);
let response_value = serde_json::to_value(build_response("q1", vec![item], 1, 5)).unwrap();
for value in [&item_value, &response_value] {
assert_absent_fields(value, &["score", "rank", "reputation", "weight", "rating"]);
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_aggregate_appeals() {
let relay_url = spawn_relay().await;
let http = client();
let sender = Keypair::generate();
let member = Keypair::generate();
let dispute = Envelope::new(
&sender,
"dispute",
serde_json::json!({"about": "counter", "reason": "unfair"}),
);
assert_eq!(
publish(&http, &relay_url, &dispute).await.status(),
reqwest::StatusCode::BAD_REQUEST
);
assert_eq!(
unicast(&http, &relay_url, &member.public_hex(), &dispute)
.await
.status(),
reqwest::StatusCode::BAD_REQUEST
);
assert_eq!(
poll(&http, &relay_url, &member.public_hex(), 30)
.await
.status(),
reqwest::StatusCode::NO_CONTENT
);
let aggregate = Envelope::new(
&sender,
TYPE_AGGREGATE,
serde_json::to_value(AggregateBody {
period: "2026-03".to_string(),
sent: 1,
passed: 1,
cited: 0,
})
.unwrap(),
);
assert_eq!(
unicast(&http, &relay_url, &member.public_hex(), &aggregate)
.await
.status(),
reqwest::StatusCode::OK
);
let body = serde_json::to_value(AggregateBody {
period: "2026-03".to_string(),
sent: 1,
passed: 1,
cited: 0,
})
.unwrap();
assert_exact_keys(&body, &["period", "sent", "passed", "cited"]);
assert_absent_fields(&body, &["appeal", "dispute", "complaint", "sanction"]);
for route in ["/v1/appeal", "/v1/dispute"] {
let response = http
.get(format!("{relay_url}{route}"))
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::NOT_FOUND);
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_topic_channels() {
let query = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
assert_exact_keys(&query, &["qid", "text", "entities", "budget"]);
assert_absent_fields(
&query,
&[
"topic", "channel", "taxonomy", "category", "routing", "topic_id",
],
);
let relay_url = spawn_relay().await;
let http = client();
let alice = Keypair::generate();
let bob = Keypair::generate();
assert_eq!(
poll(&http, &relay_url, &alice.public_hex(), 30)
.await
.status(),
reqwest::StatusCode::NO_CONTENT
);
assert_eq!(
poll(&http, &relay_url, &bob.public_hex(), 30)
.await
.status(),
reqwest::StatusCode::NO_CONTENT
);
let envelope = Envelope::new(
&alice,
TYPE_QUERY,
serde_json::json!({"qid": "q1", "text": "rust"}),
);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
for member in [&alice, &bob] {
let response = poll(&http, &relay_url, &member.public_hex(), 300).await;
let payload: Value = response.json().await.unwrap();
assert_eq!(
payload
.get("messages")
.and_then(Value::as_array)
.map(Vec::len),
Some(1),
"receiver-side filtering expected, got {payload}"
);
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_broadcast_responses() {
let relay_url = spawn_relay().await;
let http = client();
let key = Keypair::generate();
for msg_type in [TYPE_RESPONSE, "evidence", "commons", "broadcast_response"] {
let envelope = Envelope::new(&key, msg_type, serde_json::json!({}));
assert_eq!(
publish(&http, &relay_url, &envelope).await.status(),
reqwest::StatusCode::BAD_REQUEST,
"relay accepted broadcast {msg_type}"
);
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_normative_query_canonical_form() {
let root = tempfile::tempdir().unwrap();
let docs = root.path().join("docs");
fs::create_dir_all(&docs).unwrap();
fs::write(docs.join("alpha.txt"), "rust ownership and borrowing").unwrap();
let config = config_for(&root.path().join("alice"), "alice", "http://127.0.0.1:1");
{
let index = LocalIndex::open(&config.index_dir()).unwrap();
index
.add_collection(&collection("docs", &docs, true, "full"))
.unwrap();
}
let node = Node::start(config).await.unwrap();
let text = "rust?! ownership/borrowing: \"quotes\" ünïcode {braces} #tag";
let outcome = node::control_query(
&format!("http://{}", node.addr),
text,
Some(5),
Some(100),
false,
)
.await
.unwrap();
assert_eq!(outcome.get("text").and_then(Value::as_str), Some(text));
assert_eq!(
outcome.pointer("/local/total").and_then(Value::as_u64),
Some(1),
"arbitrary phrasing should match lexically: {outcome}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_supply_announce_firehose() {
let relay_url = spawn_relay().await;
let http = client();
let key = Keypair::generate();
for msg_type in ["announce", "supply", "documents", "publish"] {
let envelope = Envelope::new(&key, msg_type, serde_json::json!({}));
assert_eq!(
publish(&http, &relay_url, &envelope).await.status(),
reqwest::StatusCode::BAD_REQUEST,
"relay accepted supply announcement {msg_type}"
);
}
for route in ["/v1/announce", "/v1/supply"] {
let response = http
.get(format!("{relay_url}{route}"))
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::NOT_FOUND);
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn no_durable_replayable_broadcast_stream() {
let relay_url = spawn_relay().await;
let http = client();
let key = Keypair::generate();
let envelope = Envelope::new(
&key,
TYPE_QUERY,
serde_json::json!({"qid": "q1", "text": "rust"}),
);
assert!(
publish(&http, &relay_url, &envelope)
.await
.status()
.is_success()
);
let response = poll(&http, &relay_url, &key.public_hex(), 300).await;
let payload: Value = response.json().await.unwrap();
assert_eq!(
payload
.get("messages")
.and_then(Value::as_array)
.map(Vec::len),
Some(1)
);
assert_eq!(
poll(&http, &relay_url, &key.public_hex(), 50)
.await
.status(),
reqwest::StatusCode::NO_CONTENT,
"relay replayed a drained message"
);
for route in ["/v1/history", "/v1/replay"] {
let response = http
.get(format!("{relay_url}{route}"))
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::NOT_FOUND);
}
}
+51
View File
@@ -0,0 +1,51 @@
use std::fs;
use std::time::Instant;
use frxd::index::{Collection, LocalIndex, response_items};
use frxd::message::{EXPOSURE_FULL, build_response};
#[test]
fn thousand_file_corpus_is_searchable_and_honest() {
let temp = tempfile::tempdir().unwrap();
let docs = temp.path().join("corpus");
fs::create_dir_all(&docs).unwrap();
let total = 1000;
for i in 0..total {
let part = docs.join(format!("part{:02}", i / 100));
fs::create_dir_all(&part).unwrap();
fs::write(
part.join(format!("doc{i:04}.txt")),
format!("commonneedle unique{i} rust document number {i}"),
)
.unwrap();
}
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
let start = Instant::now();
let added = index
.add_collection(&Collection {
name: "corpus".to_string(),
path: docs.display().to_string(),
shared: true,
exposure: EXPOSURE_FULL.to_string(),
})
.unwrap();
println!("indexed {added} documents in {:?}", start.elapsed());
assert_eq!(added, total);
assert_eq!(index.doc_count(), total as u64);
let start = Instant::now();
let (hits, matches) = index.search("commonneedle", 10, true).unwrap();
println!("query matched {matches} documents in {:?}", start.elapsed());
assert_eq!(matches, total as u64);
assert_eq!(hits.len(), 10);
let response = build_response("scale", response_items(&hits), matches, 10);
assert!(response.truncated);
assert_eq!(response.more_available, (total - 10) as u64);
assert!(serde_json::to_string(&response).unwrap().len() > 0);
let (hits, matches) = index.search("unique777", 10, true).unwrap();
assert_eq!(matches, 1);
assert!(hits[0].summary.contains("unique777"));
}