Add Phase 1 frxd implementation with conformance test suite
This commit is contained in:
@@ -0,0 +1 @@
|
|||||||
|
/target
|
||||||
@@ -1,8 +1,10 @@
|
|||||||
# AGENTS.md
|
# AGENTS.md
|
||||||
|
|
||||||
## Repo shape
|
## Repo shape
|
||||||
- Spec-only repo: `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.3) is the entire content. No code, README, manifests, tests, CI, or build tooling — don't look for or invent build/lint/test commands.
|
- `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.3) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
|
||||||
- `frxd` (Rust: tokio, tantivy, axum, ed25519-dalek) is planned in §7 but does not exist here; treat it as design intent, not current structure.
|
- `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
|
||||||
|
- Commands: `cargo build`, `cargo test` (62 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
|
||||||
|
- E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.
|
||||||
|
|
||||||
## Editing the spec
|
## Editing the spec
|
||||||
- Read all of `rfc.txt` before editing; it is the sole source of truth and is deliberately terse.
|
- Read all of `rfc.txt` before editing; it is the sole source of truth and is deliberately terse.
|
||||||
@@ -12,10 +14,26 @@
|
|||||||
- §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something.
|
- §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something.
|
||||||
- Use the spec's vocabulary — member/querier/responder, citations/receipts/aggregates, source/enrichment members — not client/server or search-engine terms.
|
- Use the spec's vocabulary — member/querier/responder, citations/receipts/aggregates, source/enrichment members — not client/server or search-engine terms.
|
||||||
|
|
||||||
|
## Implementation notes
|
||||||
|
- Envelope signing is provisional (`src/crypto.rs`): `FRX/0.3` + fields + sorted-key canonical JSON body. §10's signature canonicalization open issue is unsolved — never present this scheme as interoperable.
|
||||||
|
- Phase 1 only: no aggregates (Phase 2), no receipts/lineage/delegation (Phase 3). Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
|
||||||
|
- Relay verifies signatures, carries only `query` broadcasts, holds no history (queue drained on poll), and returns 429 + Retry-After under backpressure — never silent drops.
|
||||||
|
- Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits ordered results with honest `truncated`/`more_available` and no scores (I6).
|
||||||
|
- Index layout: Tantivy at `<data_dir>/index`, collections manifest at `<data_dir>/collections.toml`; `exposure` (metadata|full) gates whether `content` is returned.
|
||||||
|
- Egress checks live in the responder path (`src/node.rs` `respond`), not the relay — keep private collections unreachable there.
|
||||||
|
- Appendix B is executable policy: `tests/purges.rs` has one absence test per rejected mechanism (11 rows, not the RFC's informal "nine"). Add a negative test there before ever re-proposing one, and only if the rationale is addressed.
|
||||||
|
- `add`/`reindex` reset a collection (delete by manifest `name`) before re-adding, so deleted files don't linger; collection identity is its name, and same-named collections replace each other.
|
||||||
|
- Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale).
|
||||||
|
|
||||||
|
## Known gaps (Phase 2/3, intentional — don't fake them)
|
||||||
|
- No aggregate serving/dashboard (RFC §9 conformance is partial without it), no directory watching (new files need `reindex`), no receipts/lineage/delegation, no TLS, no consumer admission/relay discovery.
|
||||||
|
- Node query dedup is by `qid` only; there is no envelope replay/nonce window (RFC doesn't require one).
|
||||||
|
|
||||||
## Technical plans (deliberately not in the RFC)
|
## Technical plans (deliberately not in the RFC)
|
||||||
- Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface.
|
- Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface.
|
||||||
- Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as enrichment members / backfill seeding. RFC §6 and Appendix A already name both as example derived corpora, so no new mechanisms are required; enrichment members are metadata-only exposure and sit outside the citation market.
|
- Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as enrichment members / backfill seeding. RFC §6 and Appendix A already name both as example derived corpora, so no new mechanisms are required; enrichment members are metadata-only exposure and sit outside the citation market.
|
||||||
- Phase 1 remains the minimal two-node query/response demo (§7); the enrichment demo layers on top of it.
|
- Phase 1 (two-node query/response) is built and tested; the enrichment demo layers on top of it.
|
||||||
- Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate.
|
- Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate.
|
||||||
- frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher".
|
- frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher".
|
||||||
|
- Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role.
|
||||||
- Matching accuracy is a project-health concern: start lexical (Tantivy), plan a hybrid cheap lexical gate + optional local embedding rerank (two-stage ingestion, Appendix A); embedding model stays local and replaceable (I2/I5).
|
- Matching accuracy is a project-health concern: start lexical (Tantivy), plan a hybrid cheap lexical gate + optional local embedding rerank (two-stage ingestion, Appendix A); embedding model stays local and replaceable (I2/I5).
|
||||||
|
|||||||
Generated
+3035
File diff suppressed because it is too large
Load Diff
+22
@@ -0,0 +1,22 @@
|
|||||||
|
[package]
|
||||||
|
name = "frxd"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2024"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
anyhow = "1.0.104"
|
||||||
|
axum = "0.8.9"
|
||||||
|
chrono = { version = "0.4.45", default-features = false, features = ["clock", "std"] }
|
||||||
|
clap = { version = "4.6.7", features = ["derive"] }
|
||||||
|
ed25519-dalek = { version = "3.0.0", features = ["rand_core"] }
|
||||||
|
hex = "0.4.3"
|
||||||
|
rand = "0.8"
|
||||||
|
reqwest = { version = "0.13.5", default-features = false, features = ["json", "rustls", "http2"] }
|
||||||
|
serde = { version = "1.0.229", features = ["derive"] }
|
||||||
|
serde_json = "1.0.151"
|
||||||
|
tantivy = "0.26.2"
|
||||||
|
tokio = { version = "1.53.1", features = ["full"] }
|
||||||
|
toml = "1.1.6"
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
tempfile = "3.27.0"
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
use clap::{Parser, Subcommand};
|
||||||
|
use frxd::commands;
|
||||||
|
|
||||||
|
#[derive(Parser)]
|
||||||
|
#[command(
|
||||||
|
name = "frx",
|
||||||
|
version,
|
||||||
|
about = "FRX client — local search and broadcast queries"
|
||||||
|
)]
|
||||||
|
struct Cli {
|
||||||
|
#[arg(long, global = true, default_value = "frxd.toml")]
|
||||||
|
config: PathBuf,
|
||||||
|
#[command(subcommand)]
|
||||||
|
command: Command,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Subcommand)]
|
||||||
|
enum Command {
|
||||||
|
Search {
|
||||||
|
text: String,
|
||||||
|
#[arg(long, default_value_t = 10)]
|
||||||
|
limit: usize,
|
||||||
|
},
|
||||||
|
Query {
|
||||||
|
text: String,
|
||||||
|
#[arg(long)]
|
||||||
|
max_results: Option<usize>,
|
||||||
|
#[arg(long)]
|
||||||
|
timeout_ms: Option<u64>,
|
||||||
|
#[arg(long)]
|
||||||
|
local_only: bool,
|
||||||
|
},
|
||||||
|
Status,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
let cli = Cli::parse();
|
||||||
|
match cli.command {
|
||||||
|
Command::Search { text, limit } => commands::search(&cli.config, &text, limit).await?,
|
||||||
|
Command::Query {
|
||||||
|
text,
|
||||||
|
max_results,
|
||||||
|
timeout_ms,
|
||||||
|
local_only,
|
||||||
|
} => {
|
||||||
|
commands::query(&cli.config, &text, max_results, timeout_ms, !local_only).await?;
|
||||||
|
}
|
||||||
|
Command::Status => commands::status(&cli.config).await?,
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
+115
@@ -0,0 +1,115 @@
|
|||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
|
||||||
|
use crate::config::Config;
|
||||||
|
use crate::index::{Collection, LocalIndex, load_collections, save_collections};
|
||||||
|
use crate::message::{EXPOSURE_FULL, EXPOSURE_METADATA};
|
||||||
|
use crate::node;
|
||||||
|
use crate::render;
|
||||||
|
|
||||||
|
pub fn add(
|
||||||
|
config_path: &Path,
|
||||||
|
path: &Path,
|
||||||
|
name: Option<String>,
|
||||||
|
shared: bool,
|
||||||
|
exposure: &str,
|
||||||
|
) -> Result<()> {
|
||||||
|
let config = Config::load(config_path)?;
|
||||||
|
let index = LocalIndex::open(&config.index_dir())?;
|
||||||
|
let name = name.unwrap_or_else(|| {
|
||||||
|
path.file_name()
|
||||||
|
.and_then(|n| n.to_str())
|
||||||
|
.unwrap_or("collection")
|
||||||
|
.to_string()
|
||||||
|
});
|
||||||
|
let exposure = if exposure == EXPOSURE_FULL {
|
||||||
|
EXPOSURE_FULL
|
||||||
|
} else {
|
||||||
|
EXPOSURE_METADATA
|
||||||
|
};
|
||||||
|
let collection = Collection {
|
||||||
|
name: name.clone(),
|
||||||
|
path: path
|
||||||
|
.canonicalize()
|
||||||
|
.with_context(|| format!("resolving {}", path.display()))?
|
||||||
|
.display()
|
||||||
|
.to_string(),
|
||||||
|
shared,
|
||||||
|
exposure: exposure.to_string(),
|
||||||
|
};
|
||||||
|
let added = index.add_collection(&collection)?;
|
||||||
|
let manifest = config.collections_path();
|
||||||
|
let mut collections = load_collections(&manifest)?;
|
||||||
|
collections.retain(|existing| existing.name != name);
|
||||||
|
collections.push(collection);
|
||||||
|
save_collections(&manifest, &collections)?;
|
||||||
|
println!(
|
||||||
|
"indexed {added} file(s) from {} as collection '{name}' (shared={shared}, exposure={exposure})",
|
||||||
|
path.display()
|
||||||
|
);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn reindex(config_path: &Path) -> Result<()> {
|
||||||
|
let config = Config::load(config_path)?;
|
||||||
|
let index = LocalIndex::open(&config.index_dir())?;
|
||||||
|
let collections = load_collections(&config.collections_path())?;
|
||||||
|
if collections.is_empty() {
|
||||||
|
println!("no collections registered; use `frxd add <path>`");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
for collection in collections {
|
||||||
|
let added = index.add_collection(&collection)?;
|
||||||
|
println!(
|
||||||
|
"collection '{}' ({}): {added} file(s)",
|
||||||
|
collection.name, collection.path
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn search(config_path: &Path, text: &str, limit: usize) -> Result<()> {
|
||||||
|
let config = Config::load(config_path)?;
|
||||||
|
let index = LocalIndex::open(&config.index_dir())?;
|
||||||
|
let (hits, total) = index.search(text, limit, false)?;
|
||||||
|
render::local_hits(&hits, total);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn query(
|
||||||
|
config_path: &Path,
|
||||||
|
text: &str,
|
||||||
|
max_results: Option<usize>,
|
||||||
|
timeout_ms: Option<u64>,
|
||||||
|
network: bool,
|
||||||
|
) -> Result<()> {
|
||||||
|
let config = Config::load(config_path)?;
|
||||||
|
let base = format!("http://{}", config.node.listen);
|
||||||
|
let outcome = node::control_query(&base, text, max_results, timeout_ms, network).await?;
|
||||||
|
render::query_outcome(&outcome);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn status(config_path: &Path) -> Result<()> {
|
||||||
|
let config = Config::load(config_path)?;
|
||||||
|
let base = format!("http://{}", config.node.listen);
|
||||||
|
let client = reqwest::Client::builder()
|
||||||
|
.timeout(std::time::Duration::from_secs(2))
|
||||||
|
.build()?;
|
||||||
|
match client.get(format!("{base}/v1/local/status")).send().await {
|
||||||
|
Ok(response) if response.status().is_success() => {
|
||||||
|
let value: serde_json::Value = response.json().await?;
|
||||||
|
println!("{}", serde_json::to_string_pretty(&value)?);
|
||||||
|
}
|
||||||
|
Ok(response) => println!("node at {base} responded {}", response.status()),
|
||||||
|
Err(_) => {
|
||||||
|
println!("node not running at {base}");
|
||||||
|
println!("config: {}", config_path.display());
|
||||||
|
println!("name: {}", config.node.name);
|
||||||
|
println!("data dir: {}", config.data_dir().display());
|
||||||
|
println!("relays: {:?}", config.node.relays);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
+151
@@ -0,0 +1,151 @@
|
|||||||
|
use std::fs;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
use crate::crypto::Keypair;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Config {
|
||||||
|
pub node: NodeSection,
|
||||||
|
#[serde(default)]
|
||||||
|
pub query: QuerySection,
|
||||||
|
#[serde(default)]
|
||||||
|
pub index: IndexSection,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct NodeSection {
|
||||||
|
pub name: String,
|
||||||
|
pub listen: String,
|
||||||
|
pub relays: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
pub trusted_keys: Vec<String>,
|
||||||
|
#[serde(default = "default_true")]
|
||||||
|
pub responder: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct QuerySection {
|
||||||
|
#[serde(default = "default_max_results")]
|
||||||
|
pub max_results: usize,
|
||||||
|
#[serde(default = "default_timeout_ms")]
|
||||||
|
pub timeout_ms: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for QuerySection {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
max_results: default_max_results(),
|
||||||
|
timeout_ms: default_timeout_ms(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct IndexSection {
|
||||||
|
#[serde(default = "default_data_dir")]
|
||||||
|
pub data_dir: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for IndexSection {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
data_dir: default_data_dir(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_true() -> bool {
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_max_results() -> usize {
|
||||||
|
5
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_timeout_ms() -> u64 {
|
||||||
|
2000
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_data_dir() -> String {
|
||||||
|
"./frx-data".to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Config {
|
||||||
|
pub fn new(name: &str, listen: &str, relay: &str, data_dir: &str) -> Self {
|
||||||
|
Self {
|
||||||
|
node: NodeSection {
|
||||||
|
name: name.to_string(),
|
||||||
|
listen: listen.to_string(),
|
||||||
|
relays: vec![relay.to_string()],
|
||||||
|
trusted_keys: Vec::new(),
|
||||||
|
responder: true,
|
||||||
|
},
|
||||||
|
query: QuerySection::default(),
|
||||||
|
index: IndexSection {
|
||||||
|
data_dir: data_dir.to_string(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn load(path: &Path) -> Result<Self> {
|
||||||
|
let raw = fs::read_to_string(path)
|
||||||
|
.with_context(|| format!("reading config {}", path.display()))?;
|
||||||
|
toml::from_str(&raw).with_context(|| format!("parsing config {}", path.display()))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn save(&self, path: &Path) -> Result<()> {
|
||||||
|
if let Some(parent) = path.parent() {
|
||||||
|
if !parent.as_os_str().is_empty() {
|
||||||
|
fs::create_dir_all(parent)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fs::write(path, toml::to_string_pretty(self)?)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn data_dir(&self) -> PathBuf {
|
||||||
|
PathBuf::from(&self.index.data_dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn index_dir(&self) -> PathBuf {
|
||||||
|
self.data_dir().join("index")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn key_path(&self) -> PathBuf {
|
||||||
|
self.data_dir().join("key.hex")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn collections_path(&self) -> PathBuf {
|
||||||
|
self.data_dir().join("collections.toml")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn load_key(&self) -> Result<Keypair> {
|
||||||
|
let raw = fs::read_to_string(self.key_path())
|
||||||
|
.with_context(|| format!("reading key {}", self.key_path().display()))?;
|
||||||
|
Keypair::from_hex(&raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn save_key(&self, key: &Keypair) -> Result<()> {
|
||||||
|
fs::create_dir_all(self.data_dir())?;
|
||||||
|
let path = self.key_path();
|
||||||
|
fs::write(&path, key.to_hex())?;
|
||||||
|
set_private_permissions(&path)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn set_private_permissions(path: &Path) -> Result<()> {
|
||||||
|
#[cfg(unix)]
|
||||||
|
{
|
||||||
|
use std::os::unix::fs::PermissionsExt;
|
||||||
|
fs::set_permissions(path, fs::Permissions::from_mode(0o600))?;
|
||||||
|
}
|
||||||
|
#[cfg(not(unix))]
|
||||||
|
{
|
||||||
|
let _ = path;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
+166
@@ -0,0 +1,166 @@
|
|||||||
|
use crate::PROTOCOL;
|
||||||
|
use crate::message::Envelope;
|
||||||
|
use anyhow::{Context, Result, anyhow};
|
||||||
|
use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey};
|
||||||
|
use rand::RngCore;
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
pub struct Keypair {
|
||||||
|
signing: SigningKey,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Keypair {
|
||||||
|
pub fn generate() -> Self {
|
||||||
|
let mut bytes = [0u8; 32];
|
||||||
|
rand::thread_rng().fill_bytes(&mut bytes);
|
||||||
|
Self {
|
||||||
|
signing: SigningKey::from_bytes(&bytes),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn from_hex(s: &str) -> Result<Self> {
|
||||||
|
let raw = hex::decode(s.trim()).context("key is not hex")?;
|
||||||
|
let bytes: [u8; 32] = raw
|
||||||
|
.as_slice()
|
||||||
|
.try_into()
|
||||||
|
.map_err(|_| anyhow!("key must be 32 bytes"))?;
|
||||||
|
Ok(Self {
|
||||||
|
signing: SigningKey::from_bytes(&bytes),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn to_hex(&self) -> String {
|
||||||
|
hex::encode(self.signing.to_bytes())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn public_hex(&self) -> String {
|
||||||
|
hex::encode(self.signing.verifying_key().to_bytes())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn verifying_key(&self) -> VerifyingKey {
|
||||||
|
self.signing.verifying_key()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn sign(&self, bytes: &[u8]) -> String {
|
||||||
|
hex::encode(self.signing.sign(bytes).to_bytes())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn random_nonce() -> String {
|
||||||
|
let mut bytes = [0u8; 16];
|
||||||
|
rand::thread_rng().fill_bytes(&mut bytes);
|
||||||
|
hex::encode(bytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn random_id() -> String {
|
||||||
|
let mut bytes = [0u8; 8];
|
||||||
|
rand::thread_rng().fill_bytes(&mut bytes);
|
||||||
|
hex::encode(bytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn canonical_json(value: &Value) -> String {
|
||||||
|
let mut out = String::new();
|
||||||
|
write_canonical(value, &mut out);
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
fn write_canonical(value: &Value, out: &mut String) {
|
||||||
|
match value {
|
||||||
|
Value::Null => out.push_str("null"),
|
||||||
|
Value::Bool(b) => out.push_str(if *b { "true" } else { "false" }),
|
||||||
|
Value::Number(n) => out.push_str(&n.to_string()),
|
||||||
|
Value::String(s) => out.push_str(&serde_json::to_string(s).expect("string serializes")),
|
||||||
|
Value::Array(items) => {
|
||||||
|
out.push('[');
|
||||||
|
for (i, item) in items.iter().enumerate() {
|
||||||
|
if i > 0 {
|
||||||
|
out.push(',');
|
||||||
|
}
|
||||||
|
write_canonical(item, out);
|
||||||
|
}
|
||||||
|
out.push(']');
|
||||||
|
}
|
||||||
|
Value::Object(map) => {
|
||||||
|
let mut keys: Vec<&String> = map.keys().collect();
|
||||||
|
keys.sort();
|
||||||
|
out.push('{');
|
||||||
|
for (i, key) in keys.iter().enumerate() {
|
||||||
|
if i > 0 {
|
||||||
|
out.push(',');
|
||||||
|
}
|
||||||
|
out.push_str(&serde_json::to_string(key).expect("key serializes"));
|
||||||
|
out.push(':');
|
||||||
|
write_canonical(&map[*key], out);
|
||||||
|
}
|
||||||
|
out.push('}');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn signing_bytes(envelope: &Envelope) -> Vec<u8> {
|
||||||
|
format!(
|
||||||
|
"{}\n{}\n{}\n{}\n{}\n{}",
|
||||||
|
PROTOCOL,
|
||||||
|
envelope.msg_type,
|
||||||
|
envelope.from,
|
||||||
|
envelope.ts,
|
||||||
|
envelope.nonce,
|
||||||
|
canonical_json(&envelope.body)
|
||||||
|
)
|
||||||
|
.into_bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn verify_envelope(envelope: &Envelope) -> Result<()> {
|
||||||
|
let key_bytes = hex::decode(&envelope.from).context("from is not hex")?;
|
||||||
|
let key_bytes: [u8; 32] = key_bytes
|
||||||
|
.as_slice()
|
||||||
|
.try_into()
|
||||||
|
.map_err(|_| anyhow!("from must be a 32-byte ed25519 public key"))?;
|
||||||
|
let verifying_key =
|
||||||
|
VerifyingKey::from_bytes(&key_bytes).map_err(|e| anyhow!("bad public key: {e}"))?;
|
||||||
|
let sig_bytes = hex::decode(&envelope.sig).context("sig is not hex")?;
|
||||||
|
let sig_bytes: [u8; 64] = sig_bytes
|
||||||
|
.as_slice()
|
||||||
|
.try_into()
|
||||||
|
.map_err(|_| anyhow!("sig must be 64 bytes"))?;
|
||||||
|
let signature = Signature::from_bytes(&sig_bytes);
|
||||||
|
verifying_key
|
||||||
|
.verify_strict(&signing_bytes(envelope), &signature)
|
||||||
|
.map_err(|_| anyhow!("signature verification failed"))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn now_ts() -> u64 {
|
||||||
|
std::time::SystemTime::now()
|
||||||
|
.duration_since(std::time::UNIX_EPOCH)
|
||||||
|
.map(|d| d.as_secs())
|
||||||
|
.unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use serde_json::json;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn canonical_json_sorts_keys_recursively() {
|
||||||
|
let a = json!({"b": 1, "a": {"d": [3, 2], "c": "x"}});
|
||||||
|
let b = json!({"a": {"c": "x", "d": [3, 2]}, "b": 1});
|
||||||
|
assert_eq!(canonical_json(&a), canonical_json(&b));
|
||||||
|
assert_eq!(canonical_json(&a), r#"{"a":{"c":"x","d":[3,2]},"b":1}"#);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn envelope_sign_verify_roundtrip() {
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let env = Envelope::new(&key, crate::message::TYPE_QUERY, json!({"text": "hello"}));
|
||||||
|
verify_envelope(&env).unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn tampered_body_fails_verification() {
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let mut env = Envelope::new(&key, crate::message::TYPE_QUERY, json!({"text": "hello"}));
|
||||||
|
env.body = json!({"text": "hello", "extra": true});
|
||||||
|
assert!(verify_envelope(&env).is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
+242
@@ -0,0 +1,242 @@
|
|||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::time::SystemTime;
|
||||||
|
|
||||||
|
use chrono::{DateTime, SecondsFormat, Utc};
|
||||||
|
|
||||||
|
pub const MAX_BODY_BYTES: usize = 1_000_000;
|
||||||
|
|
||||||
|
pub struct Extracted {
|
||||||
|
pub title: String,
|
||||||
|
pub body: String,
|
||||||
|
pub published: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn supported(path: &Path) -> bool {
|
||||||
|
match path.extension().and_then(|e| e.to_str()) {
|
||||||
|
Some(ext) => matches!(
|
||||||
|
ext.to_ascii_lowercase().as_str(),
|
||||||
|
"txt" | "md" | "markdown" | "html" | "htm" | "rst" | "log"
|
||||||
|
),
|
||||||
|
None => false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn extract_file(path: &Path) -> Option<Extracted> {
|
||||||
|
if !supported(path) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let raw = fs::read_to_string(path).ok()?;
|
||||||
|
let ext = path
|
||||||
|
.extension()
|
||||||
|
.and_then(|e| e.to_str())
|
||||||
|
.unwrap_or("")
|
||||||
|
.to_ascii_lowercase();
|
||||||
|
let body = if ext == "html" || ext == "htm" {
|
||||||
|
strip_html(&raw)
|
||||||
|
} else {
|
||||||
|
collapse_blank_lines(&raw)
|
||||||
|
};
|
||||||
|
let body = truncate_chars(&body, MAX_BODY_BYTES);
|
||||||
|
let title = if ext == "html" || ext == "htm" {
|
||||||
|
html_title(&raw).unwrap_or_else(|| fallback_title(path, &body))
|
||||||
|
} else {
|
||||||
|
markdown_title(&raw).unwrap_or_else(|| fallback_title(path, &body))
|
||||||
|
};
|
||||||
|
let published = fs::metadata(path)
|
||||||
|
.and_then(|m| m.modified())
|
||||||
|
.ok()
|
||||||
|
.map(format_time)
|
||||||
|
.unwrap_or_default();
|
||||||
|
Some(Extracted {
|
||||||
|
title: truncate_chars(title.trim(), 200),
|
||||||
|
body,
|
||||||
|
published,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn summary_of(body: &str) -> String {
|
||||||
|
truncate_chars(body.trim(), 300).replace('\n', " ")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fallback_title(path: &Path, body: &str) -> String {
|
||||||
|
if let Some(line) = body.lines().map(str::trim).find(|l| !l.is_empty()) {
|
||||||
|
if line.chars().count() > 3 {
|
||||||
|
return truncate_chars(line, 120);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
path.file_stem()
|
||||||
|
.and_then(|s| s.to_str())
|
||||||
|
.unwrap_or("untitled")
|
||||||
|
.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn markdown_title(raw: &str) -> Option<String> {
|
||||||
|
raw.lines()
|
||||||
|
.map(str::trim)
|
||||||
|
.find(|l| l.starts_with("# "))
|
||||||
|
.map(|l| l.trim_start_matches('#').trim().to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn html_title(raw: &str) -> Option<String> {
|
||||||
|
let lower = raw.to_ascii_lowercase();
|
||||||
|
let start = lower.find("<title")?;
|
||||||
|
let open_end = lower[start..].find('>')? + start + 1;
|
||||||
|
let end = lower[open_end..].find("</title>")? + open_end;
|
||||||
|
let title = decode_entities(raw[open_end..end].trim());
|
||||||
|
if title.is_empty() { None } else { Some(title) }
|
||||||
|
}
|
||||||
|
|
||||||
|
fn strip_html(raw: &str) -> String {
|
||||||
|
let mut out = String::with_capacity(raw.len() / 2);
|
||||||
|
let mut chars = raw.chars().peekable();
|
||||||
|
let mut skipping: Option<String> = None;
|
||||||
|
while let Some(c) = chars.next() {
|
||||||
|
if c == '<' {
|
||||||
|
let mut tag = String::new();
|
||||||
|
for t in chars.by_ref() {
|
||||||
|
if t == '>' {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tag.push(t);
|
||||||
|
}
|
||||||
|
let name = tag
|
||||||
|
.trim_start_matches('/')
|
||||||
|
.trim()
|
||||||
|
.split(|c: char| c.is_whitespace() || c == '/')
|
||||||
|
.next()
|
||||||
|
.unwrap_or("")
|
||||||
|
.to_ascii_lowercase();
|
||||||
|
if skipping.is_none() && (name == "script" || name == "style") {
|
||||||
|
skipping = Some(name.clone());
|
||||||
|
} else if skipping.as_deref() == Some(name.as_str())
|
||||||
|
&& tag.trim_start().starts_with('/')
|
||||||
|
{
|
||||||
|
skipping = None;
|
||||||
|
}
|
||||||
|
out.push(' ');
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if skipping.is_none() {
|
||||||
|
out.push(c);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
collapse_whitespace(&decode_entities(&out))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn collapse_whitespace(s: &str) -> String {
|
||||||
|
let mut out = String::with_capacity(s.len());
|
||||||
|
let mut last_space = false;
|
||||||
|
for c in s.chars() {
|
||||||
|
if c.is_whitespace() {
|
||||||
|
if !last_space {
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
last_space = true;
|
||||||
|
} else {
|
||||||
|
out.push(c);
|
||||||
|
last_space = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.trim().to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn collapse_blank_lines(s: &str) -> String {
|
||||||
|
let mut out = String::with_capacity(s.len());
|
||||||
|
let mut blank_run = 0;
|
||||||
|
for line in s.lines() {
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
blank_run += 1;
|
||||||
|
if blank_run > 1 {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
blank_run = 0;
|
||||||
|
}
|
||||||
|
out.push_str(line);
|
||||||
|
out.push('\n');
|
||||||
|
}
|
||||||
|
out.trim().to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn decode_entities(s: &str) -> String {
|
||||||
|
s.replace(" ", " ")
|
||||||
|
.replace("&", "&")
|
||||||
|
.replace("<", "<")
|
||||||
|
.replace(">", ">")
|
||||||
|
.replace(""", "\"")
|
||||||
|
.replace("'", "'")
|
||||||
|
.replace("'", "'")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn truncate_chars(s: &str, max: usize) -> String {
|
||||||
|
if s.len() <= max {
|
||||||
|
return s.to_string();
|
||||||
|
}
|
||||||
|
let end = s
|
||||||
|
.char_indices()
|
||||||
|
.take_while(|(i, _)| *i < max)
|
||||||
|
.last()
|
||||||
|
.map(|(i, c)| i + c.len_utf8())
|
||||||
|
.unwrap_or(0);
|
||||||
|
s[..end].to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn format_time(t: SystemTime) -> String {
|
||||||
|
let dt: DateTime<Utc> = t.into();
|
||||||
|
dt.to_rfc3339_opts(SecondsFormat::Secs, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::io::Write;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn html_is_stripped_and_title_extracted() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("page.html");
|
||||||
|
let mut f = fs::File::create(&path).unwrap();
|
||||||
|
write!(
|
||||||
|
f,
|
||||||
|
"<html><head><title>Rust & Ownership</title><style>p{{color:red}}</style></head><body><p>Hello <b>world</b></p><script>alert(1)</script></body></html>"
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let extracted = extract_file(&path).unwrap();
|
||||||
|
assert_eq!(extracted.title, "Rust & Ownership");
|
||||||
|
assert!(extracted.body.contains("Hello world"));
|
||||||
|
assert!(!extracted.body.contains("alert"));
|
||||||
|
assert!(!extracted.body.contains("color:red"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn unsupported_extensions_are_ignored() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("archive.bin");
|
||||||
|
fs::write(&path, "binary-ish").unwrap();
|
||||||
|
assert!(!supported(&path));
|
||||||
|
assert!(extract_file(&path).is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn oversized_bodies_are_truncated_on_char_boundaries() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("big.txt");
|
||||||
|
let mut body = "é".repeat(MAX_BODY_BYTES / 2 + 50);
|
||||||
|
body.push_str(" tail");
|
||||||
|
fs::write(&path, &body).unwrap();
|
||||||
|
let extracted = extract_file(&path).unwrap();
|
||||||
|
assert!(extracted.body.len() <= MAX_BODY_BYTES);
|
||||||
|
assert!(extracted.body.chars().all(|c| c == 'é'));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn markdown_heading_becomes_title() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("note.md");
|
||||||
|
fs::write(&path, "# Borrowing\n\nRules of borrowing.\n").unwrap();
|
||||||
|
let extracted = extract_file(&path).unwrap();
|
||||||
|
assert_eq!(extracted.title, "Borrowing");
|
||||||
|
assert!(extracted.body.contains("Rules of borrowing."));
|
||||||
|
}
|
||||||
|
}
|
||||||
+359
@@ -0,0 +1,359 @@
|
|||||||
|
use std::fs;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::sync::Mutex;
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use tantivy::collector::{Count, TopDocs};
|
||||||
|
use tantivy::directory::MmapDirectory;
|
||||||
|
use tantivy::query::{AllQuery, BooleanQuery, Occur, Query, QueryParser, TermQuery};
|
||||||
|
use tantivy::schema::{Field, IndexRecordOption, STORED, STRING, Schema, TEXT, Value};
|
||||||
|
use tantivy::{Index, IndexReader, IndexWriter, TantivyDocument, Term, doc};
|
||||||
|
|
||||||
|
use crate::extract::{extract_file, summary_of, supported};
|
||||||
|
use crate::message::{EXPOSURE_FULL, ResponseItem};
|
||||||
|
|
||||||
|
const WRITER_BUDGET: usize = 50_000_000;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Collection {
|
||||||
|
pub name: String,
|
||||||
|
pub path: String,
|
||||||
|
pub shared: bool,
|
||||||
|
pub exposure: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Default, Serialize, Deserialize)]
|
||||||
|
struct CollectionsFile {
|
||||||
|
#[serde(default, rename = "collection")]
|
||||||
|
collections: Vec<Collection>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn load_collections(manifest: &Path) -> Result<Vec<Collection>> {
|
||||||
|
if !manifest.exists() {
|
||||||
|
return Ok(Vec::new());
|
||||||
|
}
|
||||||
|
let raw = fs::read_to_string(manifest).context("reading collections manifest")?;
|
||||||
|
let parsed: CollectionsFile = toml::from_str(&raw).context("parsing collections manifest")?;
|
||||||
|
Ok(parsed.collections)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn save_collections(manifest: &Path, collections: &[Collection]) -> Result<()> {
|
||||||
|
if let Some(parent) = manifest.parent() {
|
||||||
|
fs::create_dir_all(parent)?;
|
||||||
|
}
|
||||||
|
let file = CollectionsFile {
|
||||||
|
collections: collections.to_vec(),
|
||||||
|
};
|
||||||
|
fs::write(manifest, toml::to_string_pretty(&file)?)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct SearchHit {
|
||||||
|
pub url: String,
|
||||||
|
pub title: String,
|
||||||
|
pub summary: String,
|
||||||
|
pub published: String,
|
||||||
|
pub exposure: String,
|
||||||
|
pub collection: String,
|
||||||
|
pub shared: bool,
|
||||||
|
pub content: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn response_items(hits: &[SearchHit]) -> Vec<ResponseItem> {
|
||||||
|
hits.iter()
|
||||||
|
.map(|hit| ResponseItem {
|
||||||
|
url: hit.url.clone(),
|
||||||
|
title: hit.title.clone(),
|
||||||
|
summary: hit.summary.clone(),
|
||||||
|
published: hit.published.clone(),
|
||||||
|
exposure: hit.exposure.clone(),
|
||||||
|
content: if hit.exposure == EXPOSURE_FULL {
|
||||||
|
hit.content.clone()
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
},
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Fields {
|
||||||
|
url: Field,
|
||||||
|
title: Field,
|
||||||
|
body: Field,
|
||||||
|
summary: Field,
|
||||||
|
published: Field,
|
||||||
|
exposure: Field,
|
||||||
|
collection: Field,
|
||||||
|
shared: Field,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct LocalIndex {
|
||||||
|
index: Index,
|
||||||
|
reader: IndexReader,
|
||||||
|
writer: Mutex<IndexWriter>,
|
||||||
|
fields: Fields,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LocalIndex {
|
||||||
|
pub fn open(dir: &Path) -> Result<Self> {
|
||||||
|
fs::create_dir_all(dir)?;
|
||||||
|
let schema = build_schema();
|
||||||
|
let index = Index::open_or_create(MmapDirectory::open(dir)?, schema)
|
||||||
|
.context("opening tantivy index")?;
|
||||||
|
let schema = index.schema();
|
||||||
|
let fields = Fields {
|
||||||
|
url: schema.get_field("url")?,
|
||||||
|
title: schema.get_field("title")?,
|
||||||
|
body: schema.get_field("body")?,
|
||||||
|
summary: schema.get_field("summary")?,
|
||||||
|
published: schema.get_field("published")?,
|
||||||
|
exposure: schema.get_field("exposure")?,
|
||||||
|
collection: schema.get_field("collection")?,
|
||||||
|
shared: schema.get_field("shared")?,
|
||||||
|
};
|
||||||
|
let reader = index.reader()?;
|
||||||
|
let writer = index.writer::<TantivyDocument>(WRITER_BUDGET)?;
|
||||||
|
Ok(Self {
|
||||||
|
index,
|
||||||
|
reader,
|
||||||
|
writer: Mutex::new(writer),
|
||||||
|
fields,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn add_collection(&self, collection: &Collection) -> Result<usize> {
|
||||||
|
let root = PathBuf::from(&collection.path);
|
||||||
|
{
|
||||||
|
let writer = self.writer.lock().expect("writer lock");
|
||||||
|
writer.delete_term(Term::from_field_text(
|
||||||
|
self.fields.collection,
|
||||||
|
&collection.name,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let mut files = Vec::new();
|
||||||
|
walk(&root, &mut files);
|
||||||
|
let mut added = 0;
|
||||||
|
for file in files {
|
||||||
|
if self.add_file(collection, &file)? {
|
||||||
|
added += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.commit()?;
|
||||||
|
Ok(added)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn add_file(&self, collection: &Collection, path: &Path) -> Result<bool> {
|
||||||
|
let Some(extracted) = extract_file(path) else {
|
||||||
|
return Ok(false);
|
||||||
|
};
|
||||||
|
let canonical = path.canonicalize().unwrap_or_else(|_| path.to_path_buf());
|
||||||
|
let url = format!("file://{}", canonical.display());
|
||||||
|
let summary = summary_of(&extracted.body);
|
||||||
|
let shared = if collection.shared { "true" } else { "false" };
|
||||||
|
let writer = self.writer.lock().expect("writer lock");
|
||||||
|
writer.delete_term(Term::from_field_text(self.fields.url, &url));
|
||||||
|
writer.add_document(doc!(
|
||||||
|
self.fields.url => url.as_str(),
|
||||||
|
self.fields.title => extracted.title.as_str(),
|
||||||
|
self.fields.body => extracted.body.as_str(),
|
||||||
|
self.fields.summary => summary.as_str(),
|
||||||
|
self.fields.published => extracted.published.as_str(),
|
||||||
|
self.fields.exposure => collection.exposure.as_str(),
|
||||||
|
self.fields.collection => collection.name.as_str(),
|
||||||
|
self.fields.shared => shared,
|
||||||
|
))?;
|
||||||
|
Ok(true)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn commit(&self) -> Result<()> {
|
||||||
|
self.writer.lock().expect("writer lock").commit()?;
|
||||||
|
self.reader.reload()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn search(
|
||||||
|
&self,
|
||||||
|
text: &str,
|
||||||
|
limit: usize,
|
||||||
|
only_shared: bool,
|
||||||
|
) -> Result<(Vec<SearchHit>, u64)> {
|
||||||
|
let limit = limit.max(1);
|
||||||
|
self.reader.reload()?;
|
||||||
|
let searcher = self.reader.searcher();
|
||||||
|
let user_query: Box<dyn Query> = if text.trim().is_empty() {
|
||||||
|
Box::new(AllQuery)
|
||||||
|
} else {
|
||||||
|
let parser =
|
||||||
|
QueryParser::for_index(&self.index, vec![self.fields.title, self.fields.body]);
|
||||||
|
let (query, _errors) = parser.parse_query_lenient(text);
|
||||||
|
query
|
||||||
|
};
|
||||||
|
let query: Box<dyn Query> = if only_shared {
|
||||||
|
let shared_term = TermQuery::new(
|
||||||
|
Term::from_field_text(self.fields.shared, "true"),
|
||||||
|
IndexRecordOption::Basic,
|
||||||
|
);
|
||||||
|
Box::new(BooleanQuery::new(vec![
|
||||||
|
(Occur::Must, user_query),
|
||||||
|
(Occur::Must, Box::new(shared_term)),
|
||||||
|
]))
|
||||||
|
} else {
|
||||||
|
user_query
|
||||||
|
};
|
||||||
|
let total = searcher.search(&*query, &Count)? as u64;
|
||||||
|
let top = searcher.search(&*query, &TopDocs::with_limit(limit).order_by_score())?;
|
||||||
|
let mut hits = Vec::with_capacity(top.len());
|
||||||
|
for (_score, address) in top {
|
||||||
|
let document: TantivyDocument = searcher.doc(address)?;
|
||||||
|
hits.push(SearchHit {
|
||||||
|
url: text_value(&document, self.fields.url),
|
||||||
|
title: text_value(&document, self.fields.title),
|
||||||
|
summary: text_value(&document, self.fields.summary),
|
||||||
|
published: text_value(&document, self.fields.published),
|
||||||
|
exposure: text_value(&document, self.fields.exposure),
|
||||||
|
collection: text_value(&document, self.fields.collection),
|
||||||
|
shared: text_value(&document, self.fields.shared) == "true",
|
||||||
|
content: Some(text_value(&document, self.fields.body)),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok((hits, total))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn doc_count(&self) -> u64 {
|
||||||
|
self.reader.searcher().num_docs()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn text_value(document: &TantivyDocument, field: Field) -> String {
|
||||||
|
document
|
||||||
|
.get_first(field)
|
||||||
|
.and_then(|value| value.as_str())
|
||||||
|
.unwrap_or("")
|
||||||
|
.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_schema() -> Schema {
|
||||||
|
let mut builder = Schema::builder();
|
||||||
|
builder.add_text_field("url", STRING | STORED);
|
||||||
|
builder.add_text_field("title", TEXT | STORED);
|
||||||
|
builder.add_text_field("body", TEXT | STORED);
|
||||||
|
builder.add_text_field("summary", STORED);
|
||||||
|
builder.add_text_field("published", STORED);
|
||||||
|
builder.add_text_field("exposure", STRING | STORED);
|
||||||
|
builder.add_text_field("collection", STRING | STORED);
|
||||||
|
builder.add_text_field("shared", STRING | STORED);
|
||||||
|
builder.build()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn walk(root: &Path, out: &mut Vec<PathBuf>) {
|
||||||
|
let Ok(entries) = fs::read_dir(root) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
for entry in entries.flatten() {
|
||||||
|
let Ok(file_type) = entry.file_type() else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if file_type.is_symlink() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = entry.file_name();
|
||||||
|
let name = name.to_string_lossy();
|
||||||
|
if file_type.is_dir() {
|
||||||
|
if name.starts_with('.') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
walk(&entry.path(), out);
|
||||||
|
} else if file_type.is_file() && supported(&entry.path()) {
|
||||||
|
out.push(entry.path());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::message::EXPOSURE_METADATA;
|
||||||
|
|
||||||
|
fn collection(name: &str, root: &Path, shared: bool, exposure: &str) -> Collection {
|
||||||
|
Collection {
|
||||||
|
name: name.to_string(),
|
||||||
|
path: root.display().to_string(),
|
||||||
|
shared,
|
||||||
|
exposure: exposure.to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shared_filter_and_exposure_are_enforced() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let shared_dir = temp.path().join("shared");
|
||||||
|
let private_dir = temp.path().join("private");
|
||||||
|
fs::create_dir_all(&shared_dir).unwrap();
|
||||||
|
fs::create_dir_all(&private_dir).unwrap();
|
||||||
|
fs::write(shared_dir.join("a.txt"), "rust ownership rules").unwrap();
|
||||||
|
fs::write(private_dir.join("b.txt"), "rust secret notes").unwrap();
|
||||||
|
|
||||||
|
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("shared", &shared_dir, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection(
|
||||||
|
"private",
|
||||||
|
&private_dir,
|
||||||
|
false,
|
||||||
|
EXPOSURE_METADATA,
|
||||||
|
))
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let (hits, total) = index.search("rust", 10, false).unwrap();
|
||||||
|
assert_eq!(total, 2);
|
||||||
|
assert_eq!(hits.len(), 2);
|
||||||
|
|
||||||
|
let (shared_hits, shared_total) = index.search("rust", 10, true).unwrap();
|
||||||
|
assert_eq!(shared_total, 1);
|
||||||
|
assert_eq!(shared_hits.len(), 1);
|
||||||
|
assert!(shared_hits[0].url.contains("shared"));
|
||||||
|
|
||||||
|
let items = response_items(&shared_hits);
|
||||||
|
assert!(items[0].content.is_some());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reindex_removes_deleted_files() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let dir = temp.path().join("corpus");
|
||||||
|
fs::create_dir_all(&dir).unwrap();
|
||||||
|
fs::write(dir.join("a.txt"), "rust one").unwrap();
|
||||||
|
fs::write(dir.join("b.txt"), "rust two").unwrap();
|
||||||
|
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("test", &dir, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(index.search("rust", 10, false).unwrap().1, 2);
|
||||||
|
|
||||||
|
fs::remove_file(dir.join("b.txt")).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("test", &dir, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(index.search("rust", 10, false).unwrap().1, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn metadata_exposure_withholds_content() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let dir = temp.path().join("corpus");
|
||||||
|
fs::create_dir_all(&dir).unwrap();
|
||||||
|
fs::write(dir.join("a.txt"), "metadata only").unwrap();
|
||||||
|
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("test", &dir, true, EXPOSURE_METADATA))
|
||||||
|
.unwrap();
|
||||||
|
let (hits, _) = index.search("metadata", 10, true).unwrap();
|
||||||
|
let items = response_items(&hits);
|
||||||
|
assert!(items[0].content.is_none());
|
||||||
|
assert_eq!(items[0].exposure, EXPOSURE_METADATA);
|
||||||
|
}
|
||||||
|
}
|
||||||
+12
@@ -0,0 +1,12 @@
|
|||||||
|
pub mod commands;
|
||||||
|
pub mod config;
|
||||||
|
pub mod crypto;
|
||||||
|
pub mod extract;
|
||||||
|
pub mod index;
|
||||||
|
pub mod message;
|
||||||
|
pub mod node;
|
||||||
|
pub mod relay;
|
||||||
|
pub mod render;
|
||||||
|
|
||||||
|
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
|
||||||
|
pub const PROTOCOL: &str = "FRX/0.3";
|
||||||
+141
@@ -0,0 +1,141 @@
|
|||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use anyhow::{Result, bail};
|
||||||
|
use clap::{Parser, Subcommand, ValueEnum};
|
||||||
|
use frxd::config::Config;
|
||||||
|
use frxd::crypto::Keypair;
|
||||||
|
use frxd::message::{EXPOSURE_FULL, EXPOSURE_METADATA};
|
||||||
|
use frxd::{commands, node, relay};
|
||||||
|
|
||||||
|
#[derive(Parser)]
|
||||||
|
#[command(
|
||||||
|
name = "frxd",
|
||||||
|
version,
|
||||||
|
about = "FRX member node — querier, responder, and local index (Draft 0.3)"
|
||||||
|
)]
|
||||||
|
struct Cli {
|
||||||
|
#[arg(long, global = true, default_value = "frxd.toml")]
|
||||||
|
config: PathBuf,
|
||||||
|
#[command(subcommand)]
|
||||||
|
command: Command,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Subcommand)]
|
||||||
|
enum Command {
|
||||||
|
Init {
|
||||||
|
#[arg(long, default_value = "member")]
|
||||||
|
name: String,
|
||||||
|
#[arg(long, default_value = "127.0.0.1:7701")]
|
||||||
|
listen: String,
|
||||||
|
#[arg(long, default_value = "http://127.0.0.1:7700")]
|
||||||
|
relay: String,
|
||||||
|
#[arg(long, default_value = "./frx-data")]
|
||||||
|
data_dir: String,
|
||||||
|
#[arg(long)]
|
||||||
|
force: bool,
|
||||||
|
},
|
||||||
|
Add {
|
||||||
|
path: PathBuf,
|
||||||
|
#[arg(long)]
|
||||||
|
name: Option<String>,
|
||||||
|
#[arg(long)]
|
||||||
|
shared: bool,
|
||||||
|
#[arg(long, value_enum, default_value_t = Exposure::Full)]
|
||||||
|
exposure: Exposure,
|
||||||
|
},
|
||||||
|
Reindex,
|
||||||
|
Search {
|
||||||
|
text: String,
|
||||||
|
#[arg(long, default_value_t = 10)]
|
||||||
|
limit: usize,
|
||||||
|
},
|
||||||
|
Query {
|
||||||
|
text: String,
|
||||||
|
#[arg(long)]
|
||||||
|
max_results: Option<usize>,
|
||||||
|
#[arg(long)]
|
||||||
|
timeout_ms: Option<u64>,
|
||||||
|
#[arg(long)]
|
||||||
|
local_only: bool,
|
||||||
|
},
|
||||||
|
Serve,
|
||||||
|
Relay {
|
||||||
|
#[arg(long, default_value = "127.0.0.1:7700")]
|
||||||
|
listen: String,
|
||||||
|
#[arg(long, default_value_t = relay::DEFAULT_CAPACITY)]
|
||||||
|
capacity: usize,
|
||||||
|
},
|
||||||
|
Status,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Copy, ValueEnum)]
|
||||||
|
enum Exposure {
|
||||||
|
Metadata,
|
||||||
|
Full,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
let cli = Cli::parse();
|
||||||
|
match cli.command {
|
||||||
|
Command::Init {
|
||||||
|
name,
|
||||||
|
listen,
|
||||||
|
relay,
|
||||||
|
data_dir,
|
||||||
|
force,
|
||||||
|
} => {
|
||||||
|
if cli.config.exists() && !force {
|
||||||
|
bail!(
|
||||||
|
"config {} already exists (use --force to overwrite)",
|
||||||
|
cli.config.display()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let config = Config::new(&name, &listen, &relay, &data_dir);
|
||||||
|
let key = Keypair::generate();
|
||||||
|
config.save_key(&key)?;
|
||||||
|
config.save(&cli.config)?;
|
||||||
|
println!("wrote config {}", cli.config.display());
|
||||||
|
println!("wrote key {}", config.key_path().display());
|
||||||
|
println!("pubkey {}", key.public_hex());
|
||||||
|
println!("data dir {}", config.data_dir().display());
|
||||||
|
}
|
||||||
|
Command::Add {
|
||||||
|
path,
|
||||||
|
name,
|
||||||
|
shared,
|
||||||
|
exposure,
|
||||||
|
} => {
|
||||||
|
let exposure = match exposure {
|
||||||
|
Exposure::Metadata => EXPOSURE_METADATA,
|
||||||
|
Exposure::Full => EXPOSURE_FULL,
|
||||||
|
};
|
||||||
|
commands::add(&cli.config, &path, name, shared, exposure)?;
|
||||||
|
}
|
||||||
|
Command::Reindex => commands::reindex(&cli.config)?,
|
||||||
|
Command::Search { text, limit } => commands::search(&cli.config, &text, limit).await?,
|
||||||
|
Command::Query {
|
||||||
|
text,
|
||||||
|
max_results,
|
||||||
|
timeout_ms,
|
||||||
|
local_only,
|
||||||
|
} => {
|
||||||
|
commands::query(&cli.config, &text, max_results, timeout_ms, !local_only).await?;
|
||||||
|
}
|
||||||
|
Command::Serve => {
|
||||||
|
let config = Config::load(&cli.config)?;
|
||||||
|
let handle = node::Node::start(config).await?;
|
||||||
|
println!("frxd listening on http://{}", handle.addr);
|
||||||
|
println!("pubkey {}", handle.pubkey);
|
||||||
|
println!("control API POST http://{}/v1/local/query", handle.addr);
|
||||||
|
tokio::signal::ctrl_c().await?;
|
||||||
|
}
|
||||||
|
Command::Relay { listen, capacity } => {
|
||||||
|
let (listener, addr) = relay::bind(&listen).await?;
|
||||||
|
println!("relay listening on http://{addr} (capacity {capacity})");
|
||||||
|
relay::run(listener, capacity).await?;
|
||||||
|
}
|
||||||
|
Command::Status => commands::status(&cli.config).await?,
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
+184
@@ -0,0 +1,184 @@
|
|||||||
|
use crate::crypto::{Keypair, now_ts, random_nonce, signing_bytes, verify_envelope};
|
||||||
|
use anyhow::{Context, Result, anyhow};
|
||||||
|
use serde::de::DeserializeOwned;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
pub const TYPE_QUERY: &str = "query";
|
||||||
|
pub const TYPE_RESPONSE: &str = "response";
|
||||||
|
pub const TYPE_AGGREGATE: &str = "aggregate";
|
||||||
|
|
||||||
|
pub const EXPOSURE_METADATA: &str = "metadata";
|
||||||
|
pub const EXPOSURE_FULL: &str = "full";
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Envelope {
|
||||||
|
#[serde(rename = "type")]
|
||||||
|
pub msg_type: String,
|
||||||
|
pub from: String,
|
||||||
|
pub ts: u64,
|
||||||
|
pub nonce: String,
|
||||||
|
pub body: Value,
|
||||||
|
pub sig: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Envelope {
|
||||||
|
pub fn new(key: &Keypair, msg_type: &str, body: Value) -> Self {
|
||||||
|
let mut envelope = Self {
|
||||||
|
msg_type: msg_type.to_string(),
|
||||||
|
from: key.public_hex(),
|
||||||
|
ts: now_ts(),
|
||||||
|
nonce: random_nonce(),
|
||||||
|
body,
|
||||||
|
sig: String::new(),
|
||||||
|
};
|
||||||
|
envelope.sig = key.sign(&signing_bytes(&envelope));
|
||||||
|
envelope
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn verify(&self) -> Result<()> {
|
||||||
|
verify_envelope(self)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn parse_body<T: DeserializeOwned>(&self) -> Result<T> {
|
||||||
|
serde_json::from_value(self.body.clone()).context("malformed body")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct Budget {
|
||||||
|
pub max_results: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct QueryBody {
|
||||||
|
pub qid: String,
|
||||||
|
pub text: String,
|
||||||
|
#[serde(default)]
|
||||||
|
pub entities: Vec<String>,
|
||||||
|
pub budget: Budget,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl QueryBody {
|
||||||
|
pub fn new(text: &str, max_results: usize) -> Self {
|
||||||
|
Self {
|
||||||
|
qid: crate::crypto::random_id(),
|
||||||
|
text: text.to_string(),
|
||||||
|
entities: Vec::new(),
|
||||||
|
budget: Budget {
|
||||||
|
max_results: max_results.clamp(1, 1000),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct ResponseItem {
|
||||||
|
pub url: String,
|
||||||
|
pub title: String,
|
||||||
|
pub summary: String,
|
||||||
|
pub published: String,
|
||||||
|
pub exposure: String,
|
||||||
|
pub content: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct ResponseBody {
|
||||||
|
pub qid: String,
|
||||||
|
pub results: Vec<ResponseItem>,
|
||||||
|
pub truncated: bool,
|
||||||
|
pub more_available: u64,
|
||||||
|
pub cursor: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn build_response(
|
||||||
|
qid: &str,
|
||||||
|
hits: Vec<ResponseItem>,
|
||||||
|
total: u64,
|
||||||
|
max_results: usize,
|
||||||
|
) -> ResponseBody {
|
||||||
|
let max_results = max_results.max(1);
|
||||||
|
let mut results = hits;
|
||||||
|
results.truncate(max_results);
|
||||||
|
let more_available = total.saturating_sub(results.len() as u64);
|
||||||
|
ResponseBody {
|
||||||
|
qid: qid.to_string(),
|
||||||
|
results,
|
||||||
|
truncated: more_available > 0,
|
||||||
|
more_available,
|
||||||
|
cursor: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct AggregateBody {
|
||||||
|
pub period: String,
|
||||||
|
pub sent: u64,
|
||||||
|
pub passed: u64,
|
||||||
|
pub cited: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn require_type(envelope: &Envelope, expected: &str) -> Result<()> {
|
||||||
|
if envelope.msg_type != expected {
|
||||||
|
return Err(anyhow!(
|
||||||
|
"expected message type {expected}, got {}",
|
||||||
|
envelope.msg_type
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use serde_json::json;
|
||||||
|
|
||||||
|
fn item(n: usize) -> ResponseItem {
|
||||||
|
ResponseItem {
|
||||||
|
url: format!("file:///doc{n}.txt"),
|
||||||
|
title: format!("doc {n}"),
|
||||||
|
summary: String::new(),
|
||||||
|
published: String::new(),
|
||||||
|
exposure: EXPOSURE_METADATA.to_string(),
|
||||||
|
content: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn max_results_is_respected() {
|
||||||
|
let response = build_response("q1", vec![item(1), item(2), item(3)], 10, 2);
|
||||||
|
assert_eq!(response.results.len(), 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn truncation_is_honest() {
|
||||||
|
let response = build_response("q1", vec![item(1), item(2), item(3)], 10, 2);
|
||||||
|
assert!(response.truncated);
|
||||||
|
assert_eq!(response.more_available, 8);
|
||||||
|
|
||||||
|
let response = build_response("q2", vec![item(1), item(2)], 2, 5);
|
||||||
|
assert!(!response.truncated);
|
||||||
|
assert_eq!(response.more_available, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn response_carries_no_scores() {
|
||||||
|
let response = build_response("q1", vec![item(1)], 1, 5);
|
||||||
|
let json = serde_json::to_string(&response).unwrap();
|
||||||
|
assert!(!json.contains("score"));
|
||||||
|
assert!(!json.contains("relevance"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_budget_is_clamped() {
|
||||||
|
let query = QueryBody::new("anything", 0);
|
||||||
|
assert_eq!(query.budget.max_results, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn envelope_rejects_wrong_type() {
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let envelope = Envelope::new(&key, TYPE_RESPONSE, json!({}));
|
||||||
|
assert!(require_type(&envelope, TYPE_QUERY).is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
+434
@@ -0,0 +1,434 @@
|
|||||||
|
use std::collections::{HashMap, HashSet};
|
||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
use std::sync::{Arc, Mutex};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use anyhow::{Context, Result, anyhow};
|
||||||
|
use axum::extract::State;
|
||||||
|
use axum::http::StatusCode;
|
||||||
|
use axum::response::{IntoResponse, Response};
|
||||||
|
use axum::routing::{get, post};
|
||||||
|
use axum::{Json, Router};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
use tokio::net::TcpListener;
|
||||||
|
use tokio::task::JoinHandle;
|
||||||
|
|
||||||
|
use crate::config::Config;
|
||||||
|
use crate::crypto::Keypair;
|
||||||
|
use crate::index::{LocalIndex, SearchHit, response_items};
|
||||||
|
use crate::message::{
|
||||||
|
Envelope, QueryBody, ResponseBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE,
|
||||||
|
build_response,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub struct Node {
|
||||||
|
pub config: Config,
|
||||||
|
pub key: Keypair,
|
||||||
|
index: LocalIndex,
|
||||||
|
seen: Mutex<HashSet<String>>,
|
||||||
|
pending: Mutex<HashMap<String, Vec<(String, ResponseBody)>>>,
|
||||||
|
client: reqwest::Client,
|
||||||
|
sent: AtomicU64,
|
||||||
|
received: AtomicU64,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct NodeHandle {
|
||||||
|
pub addr: SocketAddr,
|
||||||
|
pub pubkey: String,
|
||||||
|
pub node: Arc<Node>,
|
||||||
|
tasks: Vec<JoinHandle<()>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for NodeHandle {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
for task in &self.tasks {
|
||||||
|
task.abort();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct RemoteResponse {
|
||||||
|
pub member: String,
|
||||||
|
pub results: Vec<ResponseItem>,
|
||||||
|
pub truncated: bool,
|
||||||
|
pub more_available: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct MergedItem {
|
||||||
|
pub provenance: String,
|
||||||
|
#[serde(flatten)]
|
||||||
|
pub item: ResponseItem,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct LocalQueryOutcome {
|
||||||
|
pub qid: String,
|
||||||
|
pub text: String,
|
||||||
|
pub local: LocalPart,
|
||||||
|
pub responses: Vec<RemoteResponse>,
|
||||||
|
pub merged: Vec<MergedItem>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct LocalPart {
|
||||||
|
pub results: Vec<ResponseItem>,
|
||||||
|
pub total: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Node {
|
||||||
|
pub fn open(config: Config) -> Result<Arc<Self>> {
|
||||||
|
let key = config.load_key()?;
|
||||||
|
let index = LocalIndex::open(&config.index_dir())?;
|
||||||
|
let client = reqwest::Client::builder()
|
||||||
|
.timeout(Duration::from_secs(15))
|
||||||
|
.build()
|
||||||
|
.context("building http client")?;
|
||||||
|
Ok(Arc::new(Self {
|
||||||
|
config,
|
||||||
|
key,
|
||||||
|
index,
|
||||||
|
seen: Mutex::new(HashSet::new()),
|
||||||
|
pending: Mutex::new(HashMap::new()),
|
||||||
|
client,
|
||||||
|
sent: AtomicU64::new(0),
|
||||||
|
received: AtomicU64::new(0),
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn doc_count(&self) -> u64 {
|
||||||
|
self.index.doc_count()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn sent(&self) -> u64 {
|
||||||
|
self.sent.load(Ordering::SeqCst)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn received(&self) -> u64 {
|
||||||
|
self.received.load(Ordering::SeqCst)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn local_search(&self, text: &str, limit: usize) -> Result<(Vec<SearchHit>, u64)> {
|
||||||
|
self.index.search(text, limit, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn start(config: Config) -> Result<NodeHandle> {
|
||||||
|
let node = Node::open(config)?;
|
||||||
|
let listener = TcpListener::bind(&node.config.node.listen)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("binding {}", node.config.node.listen))?;
|
||||||
|
let addr = listener.local_addr()?;
|
||||||
|
let mut tasks = Vec::new();
|
||||||
|
for relay in node.config.node.relays.clone() {
|
||||||
|
tasks.push(tokio::spawn(poll_relay(node.clone(), relay)));
|
||||||
|
}
|
||||||
|
let app = router(node.clone());
|
||||||
|
tasks.push(tokio::spawn(async move {
|
||||||
|
if let Err(error) = axum::serve(listener, app).await {
|
||||||
|
eprintln!("control server stopped: {error}");
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
Ok(NodeHandle {
|
||||||
|
addr,
|
||||||
|
pubkey: node.key.public_hex(),
|
||||||
|
node,
|
||||||
|
tasks,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn local_query(
|
||||||
|
&self,
|
||||||
|
text: &str,
|
||||||
|
max_results: Option<usize>,
|
||||||
|
timeout_ms: Option<u64>,
|
||||||
|
network: bool,
|
||||||
|
) -> Result<LocalQueryOutcome> {
|
||||||
|
let max = max_results.unwrap_or(self.config.query.max_results).max(1);
|
||||||
|
let query = QueryBody::new(text, max);
|
||||||
|
let qid = query.qid.clone();
|
||||||
|
let (local_hits, local_total) = self.index.search(text, max, false)?;
|
||||||
|
let local_items = response_items(&local_hits);
|
||||||
|
let mut responses = Vec::new();
|
||||||
|
if network {
|
||||||
|
self.pending
|
||||||
|
.lock()
|
||||||
|
.expect("pending lock")
|
||||||
|
.insert(qid.clone(), Vec::new());
|
||||||
|
self.sent.fetch_add(1, Ordering::SeqCst);
|
||||||
|
let envelope = Envelope::new(&self.key, TYPE_QUERY, serde_json::to_value(&query)?);
|
||||||
|
self.publish(&envelope).await;
|
||||||
|
let timeout = Duration::from_millis(timeout_ms.unwrap_or(self.config.query.timeout_ms));
|
||||||
|
let deadline = Instant::now() + timeout;
|
||||||
|
while Instant::now() < deadline {
|
||||||
|
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||||
|
}
|
||||||
|
let collected = self
|
||||||
|
.pending
|
||||||
|
.lock()
|
||||||
|
.expect("pending lock")
|
||||||
|
.remove(&qid)
|
||||||
|
.unwrap_or_default();
|
||||||
|
self.received
|
||||||
|
.fetch_add(collected.len() as u64, Ordering::SeqCst);
|
||||||
|
for (member, body) in collected {
|
||||||
|
responses.push(RemoteResponse {
|
||||||
|
member,
|
||||||
|
results: body.results,
|
||||||
|
truncated: body.truncated,
|
||||||
|
more_available: body.more_available,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut merged = Vec::new();
|
||||||
|
let mut seen_urls = HashSet::new();
|
||||||
|
for item in &local_items {
|
||||||
|
if seen_urls.insert(item.url.clone()) {
|
||||||
|
merged.push(MergedItem {
|
||||||
|
provenance: "local".to_string(),
|
||||||
|
item: item.clone(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for response in &responses {
|
||||||
|
for item in &response.results {
|
||||||
|
if seen_urls.insert(item.url.clone()) {
|
||||||
|
merged.push(MergedItem {
|
||||||
|
provenance: response.member.clone(),
|
||||||
|
item: item.clone(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(LocalQueryOutcome {
|
||||||
|
qid,
|
||||||
|
text: text.to_string(),
|
||||||
|
local: LocalPart {
|
||||||
|
results: local_items,
|
||||||
|
total: local_total,
|
||||||
|
},
|
||||||
|
responses,
|
||||||
|
merged,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn publish(&self, envelope: &Envelope) -> usize {
|
||||||
|
let mut delivered = 0;
|
||||||
|
for relay in &self.config.node.relays {
|
||||||
|
let url = format!("{}/v1/publish", relay.trim_end_matches('/'));
|
||||||
|
match self.client.post(&url).json(envelope).send().await {
|
||||||
|
Ok(response) if response.status().is_success() => delivered += 1,
|
||||||
|
Ok(response) => eprintln!("relay {relay} rejected query: {}", response.status()),
|
||||||
|
Err(error) => eprintln!("relay {relay} unreachable: {error}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
delivered
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn dispatch(self: &Arc<Self>, envelope: Envelope, relay: &str) {
|
||||||
|
if envelope.verify().is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if !self.config.node.trusted_keys.is_empty()
|
||||||
|
&& !self.config.node.trusted_keys.contains(&envelope.from)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
match envelope.msg_type.as_str() {
|
||||||
|
TYPE_QUERY => {
|
||||||
|
if envelope.from == self.key.public_hex() || !self.config.node.responder {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let Ok(query) = envelope.parse_body::<QueryBody>() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
if query.text.trim().is_empty() || query.qid.is_empty() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
let mut seen = self.seen.lock().expect("seen lock");
|
||||||
|
if !seen.insert(query.qid.clone()) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if seen.len() > 10_000 {
|
||||||
|
seen.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let node = self.clone();
|
||||||
|
let relay = relay.to_string();
|
||||||
|
let querier = envelope.from.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
if let Err(error) = node.respond(&query, &querier, &relay).await {
|
||||||
|
eprintln!("responder failed: {error}");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
TYPE_RESPONSE => {
|
||||||
|
let Ok(body) = envelope.parse_body::<ResponseBody>() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let mut pending = self.pending.lock().expect("pending lock");
|
||||||
|
if let Some(list) = pending.get_mut(&body.qid) {
|
||||||
|
list.push((envelope.from.clone(), body));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
TYPE_AGGREGATE => {}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn respond(&self, query: &QueryBody, querier: &str, relay: &str) -> Result<()> {
|
||||||
|
let max = query.budget.max_results.clamp(1, 1000);
|
||||||
|
let (hits, total) = self.index.search(&query.text, max, true)?;
|
||||||
|
if hits.is_empty() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
let body = build_response(&query.qid, response_items(&hits), total, max);
|
||||||
|
let envelope = Envelope::new(&self.key, TYPE_RESPONSE, serde_json::to_value(&body)?);
|
||||||
|
let mut relays = vec![relay.to_string()];
|
||||||
|
for configured in &self.config.node.relays {
|
||||||
|
if !relays.contains(configured) {
|
||||||
|
relays.push(configured.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut last_error: Option<anyhow::Error> = None;
|
||||||
|
for candidate in relays {
|
||||||
|
let url = format!(
|
||||||
|
"{}/v1/unicast?to={}",
|
||||||
|
candidate.trim_end_matches('/'),
|
||||||
|
querier
|
||||||
|
);
|
||||||
|
match self.client.post(&url).json(&envelope).send().await {
|
||||||
|
Ok(response) if response.status().is_success() => return Ok(()),
|
||||||
|
Ok(response) => {
|
||||||
|
last_error = Some(anyhow!(
|
||||||
|
"relay {candidate} rejected response: {}",
|
||||||
|
response.status()
|
||||||
|
))
|
||||||
|
}
|
||||||
|
Err(error) => last_error = Some(anyhow!("relay {candidate} unreachable: {error}")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(last_error.unwrap_or_else(|| anyhow!("no relays configured")))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn poll_relay(node: Arc<Node>, relay: String) {
|
||||||
|
let base = relay.trim_end_matches('/').to_string();
|
||||||
|
loop {
|
||||||
|
let url = format!(
|
||||||
|
"{}/v1/poll?member={}&timeout_ms=20000",
|
||||||
|
base,
|
||||||
|
node.key.public_hex()
|
||||||
|
);
|
||||||
|
match node.client.get(&url).send().await {
|
||||||
|
Ok(response) if response.status() == reqwest::StatusCode::NO_CONTENT => continue,
|
||||||
|
Ok(response) if response.status().is_success() => {
|
||||||
|
match response.json::<Value>().await {
|
||||||
|
Ok(payload) => {
|
||||||
|
if let Some(messages) = payload.get("messages").and_then(Value::as_array) {
|
||||||
|
for message in messages {
|
||||||
|
if let Ok(envelope) =
|
||||||
|
serde_json::from_value::<Envelope>(message.clone())
|
||||||
|
{
|
||||||
|
node.dispatch(envelope, &base).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(_) => tokio::time::sleep(Duration::from_secs(1)).await,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(_) => tokio::time::sleep(Duration::from_secs(1)).await,
|
||||||
|
Err(_) => tokio::time::sleep(Duration::from_secs(1)).await,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn router(node: Arc<Node>) -> Router {
|
||||||
|
Router::new()
|
||||||
|
.route("/v1/local/query", post(local_query))
|
||||||
|
.route("/v1/local/status", get(local_status))
|
||||||
|
.with_state(node)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
pub struct LocalQueryRequest {
|
||||||
|
pub text: String,
|
||||||
|
pub max_results: Option<usize>,
|
||||||
|
pub timeout_ms: Option<u64>,
|
||||||
|
#[serde(default = "default_network")]
|
||||||
|
pub network: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_network() -> bool {
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn local_query(
|
||||||
|
State(node): State<Arc<Node>>,
|
||||||
|
Json(request): Json<LocalQueryRequest>,
|
||||||
|
) -> Response {
|
||||||
|
match node
|
||||||
|
.local_query(
|
||||||
|
&request.text,
|
||||||
|
request.max_results,
|
||||||
|
request.timeout_ms,
|
||||||
|
request.network,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(outcome) => (StatusCode::OK, Json(outcome)).into_response(),
|
||||||
|
Err(error) => (
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(json!({ "error": error.to_string() })),
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn local_status(State(node): State<Arc<Node>>) -> Response {
|
||||||
|
Json(json!({
|
||||||
|
"name": node.config.node.name,
|
||||||
|
"pubkey": node.key.public_hex(),
|
||||||
|
"listen": node.config.node.listen,
|
||||||
|
"relays": node.config.node.relays,
|
||||||
|
"responder": node.config.node.responder,
|
||||||
|
"doc_count": node.doc_count(),
|
||||||
|
"sent": node.sent(),
|
||||||
|
"received": node.received(),
|
||||||
|
}))
|
||||||
|
.into_response()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn control_query(
|
||||||
|
base: &str,
|
||||||
|
text: &str,
|
||||||
|
max_results: Option<usize>,
|
||||||
|
timeout_ms: Option<u64>,
|
||||||
|
network: bool,
|
||||||
|
) -> Result<Value> {
|
||||||
|
let client = reqwest::Client::builder()
|
||||||
|
.timeout(Duration::from_secs(30))
|
||||||
|
.build()?;
|
||||||
|
let url = format!("{}/v1/local/query", base.trim_end_matches('/'));
|
||||||
|
let response = client
|
||||||
|
.post(&url)
|
||||||
|
.json(&json!({
|
||||||
|
"text": text,
|
||||||
|
"max_results": max_results,
|
||||||
|
"timeout_ms": timeout_ms,
|
||||||
|
"network": network,
|
||||||
|
}))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("calling local node at {url} (is `frxd serve` running?)"))?;
|
||||||
|
let status = response.status();
|
||||||
|
let value: Value = response.json().await.context("parsing node response")?;
|
||||||
|
if !status.is_success() {
|
||||||
|
return Err(anyhow!("local node error {status}: {value}"));
|
||||||
|
}
|
||||||
|
Ok(value)
|
||||||
|
}
|
||||||
+191
@@ -0,0 +1,191 @@
|
|||||||
|
use std::collections::{HashMap, VecDeque};
|
||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::sync::{Arc, Mutex};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
use axum::extract::{Query, State};
|
||||||
|
use axum::http::{StatusCode, header};
|
||||||
|
use axum::response::{IntoResponse, Response};
|
||||||
|
use axum::routing::{get, post};
|
||||||
|
use axum::{Json, Router};
|
||||||
|
use serde::Deserialize;
|
||||||
|
use serde_json::json;
|
||||||
|
use tokio::net::TcpListener;
|
||||||
|
|
||||||
|
use crate::message::{Envelope, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE};
|
||||||
|
|
||||||
|
pub const DEFAULT_CAPACITY: usize = 256;
|
||||||
|
|
||||||
|
#[derive(Default)]
|
||||||
|
struct MemberQueue {
|
||||||
|
items: VecDeque<(u64, Envelope)>,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Inner {
|
||||||
|
members: HashMap<String, MemberQueue>,
|
||||||
|
seq: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct Relay {
|
||||||
|
inner: Mutex<Inner>,
|
||||||
|
capacity: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Relay {
|
||||||
|
pub fn new(capacity: usize) -> Arc<Self> {
|
||||||
|
Arc::new(Self {
|
||||||
|
inner: Mutex::new(Inner {
|
||||||
|
members: HashMap::new(),
|
||||||
|
seq: 0,
|
||||||
|
}),
|
||||||
|
capacity,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn push(&self, targets: Option<&str>, envelope: Envelope) -> Result<usize, StatusCode> {
|
||||||
|
let mut inner = self.inner.lock().expect("relay lock");
|
||||||
|
if inner
|
||||||
|
.members
|
||||||
|
.values()
|
||||||
|
.any(|q| q.items.len() >= self.capacity)
|
||||||
|
{
|
||||||
|
return Err(StatusCode::TOO_MANY_REQUESTS);
|
||||||
|
}
|
||||||
|
inner.seq += 1;
|
||||||
|
let seq = inner.seq;
|
||||||
|
match targets {
|
||||||
|
Some(member) => {
|
||||||
|
let Some(queue) = inner.members.get_mut(member) else {
|
||||||
|
return Err(StatusCode::NOT_FOUND);
|
||||||
|
};
|
||||||
|
queue.items.push_back((seq, envelope));
|
||||||
|
Ok(1)
|
||||||
|
}
|
||||||
|
None => {
|
||||||
|
let from = envelope.from.clone();
|
||||||
|
inner.members.entry(from).or_default();
|
||||||
|
let mut delivered = 0;
|
||||||
|
for queue in inner.members.values_mut() {
|
||||||
|
queue.items.push_back((seq, envelope.clone()));
|
||||||
|
delivered += 1;
|
||||||
|
}
|
||||||
|
Ok(delivered)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn router(relay: Arc<Relay>) -> Router {
|
||||||
|
Router::new()
|
||||||
|
.route("/health", get(health))
|
||||||
|
.route("/v1/publish", post(publish))
|
||||||
|
.route("/v1/unicast", post(unicast))
|
||||||
|
.route("/v1/poll", get(poll))
|
||||||
|
.with_state(relay)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn run(listener: TcpListener, capacity: usize) -> Result<()> {
|
||||||
|
let relay = Relay::new(capacity);
|
||||||
|
axum::serve(listener, router(relay)).await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn bind(addr: &str) -> Result<(TcpListener, SocketAddr)> {
|
||||||
|
let listener = TcpListener::bind(addr).await?;
|
||||||
|
let local = listener.local_addr()?;
|
||||||
|
Ok((listener, local))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn health() -> &'static str {
|
||||||
|
"ok"
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn publish(State(relay): State<Arc<Relay>>, Json(envelope): Json<Envelope>) -> Response {
|
||||||
|
if envelope.verify().is_err() {
|
||||||
|
return bad_request("invalid signature");
|
||||||
|
}
|
||||||
|
if envelope.msg_type != TYPE_QUERY {
|
||||||
|
return bad_request("relay carries broadcast queries only");
|
||||||
|
}
|
||||||
|
match relay.push(None, envelope) {
|
||||||
|
Ok(delivered) => (
|
||||||
|
StatusCode::ACCEPTED,
|
||||||
|
Json(json!({ "delivered": delivered })),
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
Err(status) => backpressure(status),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct UnicastParams {
|
||||||
|
to: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn unicast(
|
||||||
|
State(relay): State<Arc<Relay>>,
|
||||||
|
Query(params): Query<UnicastParams>,
|
||||||
|
Json(envelope): Json<Envelope>,
|
||||||
|
) -> Response {
|
||||||
|
if envelope.verify().is_err() {
|
||||||
|
return bad_request("invalid signature");
|
||||||
|
}
|
||||||
|
if envelope.msg_type != TYPE_RESPONSE && envelope.msg_type != TYPE_AGGREGATE {
|
||||||
|
return bad_request("unicast carries responses and aggregates only");
|
||||||
|
}
|
||||||
|
match relay.push(Some(¶ms.to), envelope) {
|
||||||
|
Ok(_) => (StatusCode::OK, Json(json!({ "delivered": true }))).into_response(),
|
||||||
|
Err(StatusCode::NOT_FOUND) => (
|
||||||
|
StatusCode::NOT_FOUND,
|
||||||
|
Json(json!({ "error": "member not connected" })),
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
Err(status) => backpressure(status),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct PollParams {
|
||||||
|
member: String,
|
||||||
|
timeout_ms: Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn poll(State(relay): State<Arc<Relay>>, Query(params): Query<PollParams>) -> Response {
|
||||||
|
if params.member.is_empty() || hex::decode(¶ms.member).is_err() {
|
||||||
|
return bad_request("valid member key required");
|
||||||
|
}
|
||||||
|
let timeout = Duration::from_millis(params.timeout_ms.unwrap_or(25_000).min(60_000));
|
||||||
|
let deadline = Instant::now() + timeout;
|
||||||
|
loop {
|
||||||
|
let batch: Vec<Envelope> = {
|
||||||
|
let mut inner = relay.inner.lock().expect("relay lock");
|
||||||
|
let queue = inner.members.entry(params.member.clone()).or_default();
|
||||||
|
queue
|
||||||
|
.items
|
||||||
|
.drain(..)
|
||||||
|
.map(|(_, envelope)| envelope)
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
if !batch.is_empty() {
|
||||||
|
return (StatusCode::OK, Json(json!({ "messages": batch }))).into_response();
|
||||||
|
}
|
||||||
|
if Instant::now() >= deadline {
|
||||||
|
return StatusCode::NO_CONTENT.into_response();
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bad_request(message: &str) -> Response {
|
||||||
|
(StatusCode::BAD_REQUEST, Json(json!({ "error": message }))).into_response()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backpressure(status: StatusCode) -> Response {
|
||||||
|
(
|
||||||
|
status,
|
||||||
|
[(header::RETRY_AFTER, "1")],
|
||||||
|
Json(json!({ "error": "transport backpressure" })),
|
||||||
|
)
|
||||||
|
.into_response()
|
||||||
|
}
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
use crate::index::SearchHit;
|
||||||
|
|
||||||
|
pub fn local_hits(hits: &[SearchHit], total: u64) {
|
||||||
|
println!("{total} local match(es), showing {}", hits.len());
|
||||||
|
for hit in hits {
|
||||||
|
let scope = if hit.shared { "shared" } else { "private" };
|
||||||
|
println!("- [{} / {}] {}", hit.collection, scope, hit.title);
|
||||||
|
println!(" {}", hit.url);
|
||||||
|
if !hit.summary.is_empty() {
|
||||||
|
println!(" {}", hit.summary);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn query_outcome(value: &Value) {
|
||||||
|
if let Some(qid) = value.get("qid").and_then(Value::as_str) {
|
||||||
|
println!("qid {qid}");
|
||||||
|
}
|
||||||
|
let local_total = value
|
||||||
|
.pointer("/local/total")
|
||||||
|
.and_then(Value::as_u64)
|
||||||
|
.unwrap_or(0);
|
||||||
|
println!("local: {local_total} match(es)");
|
||||||
|
if let Some(items) = value.pointer("/local/results").and_then(Value::as_array) {
|
||||||
|
for item in items {
|
||||||
|
print_item("local", item);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(responses) = value.get("responses").and_then(Value::as_array) {
|
||||||
|
println!("remote: {} response(s)", responses.len());
|
||||||
|
for response in responses {
|
||||||
|
let member = response
|
||||||
|
.get("member")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.unwrap_or("unknown");
|
||||||
|
let short = member.get(..16).unwrap_or(member);
|
||||||
|
let truncated = response
|
||||||
|
.get("truncated")
|
||||||
|
.and_then(Value::as_bool)
|
||||||
|
.unwrap_or(false);
|
||||||
|
let more = response
|
||||||
|
.get("more_available")
|
||||||
|
.and_then(Value::as_u64)
|
||||||
|
.unwrap_or(0);
|
||||||
|
println!(" from member {short}... (truncated={truncated}, more_available={more})");
|
||||||
|
if let Some(results) = response.get("results").and_then(Value::as_array) {
|
||||||
|
for item in results {
|
||||||
|
print_item(short, item);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn print_item(provenance: &str, item: &Value) {
|
||||||
|
let title = item.get("title").and_then(Value::as_str).unwrap_or("");
|
||||||
|
let url = item.get("url").and_then(Value::as_str).unwrap_or("");
|
||||||
|
let exposure = item.get("exposure").and_then(Value::as_str).unwrap_or("");
|
||||||
|
println!("- [{provenance}] {title} ({exposure})");
|
||||||
|
println!(" {url}");
|
||||||
|
}
|
||||||
+250
@@ -0,0 +1,250 @@
|
|||||||
|
mod common;
|
||||||
|
|
||||||
|
use std::fs;
|
||||||
|
use std::io::{BufRead, BufReader, Read};
|
||||||
|
use std::path::Path;
|
||||||
|
use std::process::{Child, Command, Stdio};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
fn frxd() -> Command {
|
||||||
|
Command::new(env!("CARGO_BIN_EXE_frxd"))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn frx() -> Command {
|
||||||
|
Command::new(env!("CARGO_BIN_EXE_frx"))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn init_args(
|
||||||
|
config: &Path,
|
||||||
|
name: &str,
|
||||||
|
listen_port: u16,
|
||||||
|
relay_port: u16,
|
||||||
|
data: &Path,
|
||||||
|
) -> Vec<String> {
|
||||||
|
vec![
|
||||||
|
"--config".to_string(),
|
||||||
|
config.display().to_string(),
|
||||||
|
"init".to_string(),
|
||||||
|
"--name".to_string(),
|
||||||
|
name.to_string(),
|
||||||
|
"--listen".to_string(),
|
||||||
|
format!("127.0.0.1:{listen_port}"),
|
||||||
|
"--relay".to_string(),
|
||||||
|
format!("http://127.0.0.1:{relay_port}"),
|
||||||
|
"--data-dir".to_string(),
|
||||||
|
data.display().to_string(),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn add_args(config: &Path, docs: &Path, name: &str, shared: bool) -> Vec<String> {
|
||||||
|
let mut args = vec![
|
||||||
|
"--config".to_string(),
|
||||||
|
config.display().to_string(),
|
||||||
|
"add".to_string(),
|
||||||
|
docs.display().to_string(),
|
||||||
|
"--name".to_string(),
|
||||||
|
name.to_string(),
|
||||||
|
"--exposure".to_string(),
|
||||||
|
"full".to_string(),
|
||||||
|
];
|
||||||
|
if shared {
|
||||||
|
args.push("--shared".to_string());
|
||||||
|
}
|
||||||
|
args
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_ok(args: &[String]) -> String {
|
||||||
|
let output = frxd().args(args).output().unwrap();
|
||||||
|
assert!(
|
||||||
|
output.status.success(),
|
||||||
|
"command failed: {}\n{}",
|
||||||
|
args.join(" "),
|
||||||
|
String::from_utf8_lossy(&output.stderr)
|
||||||
|
);
|
||||||
|
String::from_utf8_lossy(&output.stdout).to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Service {
|
||||||
|
child: Child,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for Service {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = self.child.kill();
|
||||||
|
let _ = self.child.wait();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn spawn_service(args: &[String], needle: &str) -> Service {
|
||||||
|
let mut child = frxd()
|
||||||
|
.args(args)
|
||||||
|
.stdout(Stdio::piped())
|
||||||
|
.stderr(Stdio::null())
|
||||||
|
.spawn()
|
||||||
|
.unwrap();
|
||||||
|
let reader = BufReader::new(child.stdout.take().unwrap());
|
||||||
|
let mut reader = reader;
|
||||||
|
let deadline = Instant::now() + Duration::from_secs(10);
|
||||||
|
loop {
|
||||||
|
let mut line = String::new();
|
||||||
|
let read = reader.read_line(&mut line).unwrap();
|
||||||
|
if read == 0 {
|
||||||
|
panic!("service exited before printing '{needle}'");
|
||||||
|
}
|
||||||
|
if line.contains(needle) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(Instant::now() < deadline, "timeout waiting for '{needle}'");
|
||||||
|
}
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
let mut sink = Vec::new();
|
||||||
|
let _ = reader.read_to_end(&mut sink);
|
||||||
|
});
|
||||||
|
Service { child }
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cli_init_add_search_status() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let config = root.path().join("frxd.toml");
|
||||||
|
let data = root.path().join("data");
|
||||||
|
let docs = root.path().join("docs");
|
||||||
|
fs::create_dir_all(&docs).unwrap();
|
||||||
|
fs::write(docs.join("note.txt"), "cli smoke rust document").unwrap();
|
||||||
|
|
||||||
|
let config_arg = config.display().to_string();
|
||||||
|
let stdout = run_ok(&init_args(&config, "alice", 0, 1, &data));
|
||||||
|
assert!(stdout.contains("pubkey"));
|
||||||
|
assert!(config.exists());
|
||||||
|
let key = data.join("key.hex");
|
||||||
|
assert!(key.exists());
|
||||||
|
#[cfg(unix)]
|
||||||
|
{
|
||||||
|
use std::os::unix::fs::PermissionsExt;
|
||||||
|
let mode = fs::metadata(&key).unwrap().permissions().mode() & 0o777;
|
||||||
|
assert_eq!(mode, 0o600, "key file must not be world readable");
|
||||||
|
}
|
||||||
|
|
||||||
|
let stdout = run_ok(&add_args(&config, &docs, "docs", true));
|
||||||
|
assert!(stdout.contains("indexed 1 file(s)"));
|
||||||
|
|
||||||
|
let stdout = run_ok(&[
|
||||||
|
"--config".to_string(),
|
||||||
|
config_arg.clone(),
|
||||||
|
"search".to_string(),
|
||||||
|
"rust".to_string(),
|
||||||
|
]);
|
||||||
|
assert!(stdout.contains("cli smoke rust document"));
|
||||||
|
|
||||||
|
let output = frx()
|
||||||
|
.args(["--config", &config_arg, "search", "rust"])
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
assert!(output.status.success());
|
||||||
|
assert!(String::from_utf8_lossy(&output.stdout).contains("cli smoke rust document"));
|
||||||
|
|
||||||
|
let stdout = run_ok(&["--config".to_string(), config_arg, "status".to_string()]);
|
||||||
|
assert!(stdout.contains("node not running"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cli_init_refuses_overwrite_without_force() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let config = root.path().join("frxd.toml");
|
||||||
|
let data = root.path().join("data");
|
||||||
|
let args = init_args(&config, "alice", 0, 1, &data);
|
||||||
|
run_ok(&args);
|
||||||
|
|
||||||
|
let output = frxd().args(&args).output().unwrap();
|
||||||
|
assert!(!output.status.success());
|
||||||
|
assert!(String::from_utf8_lossy(&output.stderr).contains("already exists"));
|
||||||
|
|
||||||
|
let mut forced = args.clone();
|
||||||
|
forced.push("--force".to_string());
|
||||||
|
run_ok(&forced);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cli_full_network_pipeline() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_port = common::free_port();
|
||||||
|
let relay_args = vec![
|
||||||
|
"relay".to_string(),
|
||||||
|
"--listen".to_string(),
|
||||||
|
format!("127.0.0.1:{relay_port}"),
|
||||||
|
];
|
||||||
|
let _relay = spawn_service(&relay_args, "relay listening");
|
||||||
|
|
||||||
|
let bob_port = common::free_port();
|
||||||
|
let alice_port = common::free_port();
|
||||||
|
let bob_config = root.path().join("bob.toml");
|
||||||
|
let alice_config = root.path().join("alice.toml");
|
||||||
|
let bob_docs = root.path().join("bob-docs");
|
||||||
|
let alice_docs = root.path().join("alice-docs");
|
||||||
|
fs::create_dir_all(&bob_docs).unwrap();
|
||||||
|
fs::create_dir_all(&alice_docs).unwrap();
|
||||||
|
fs::write(
|
||||||
|
bob_docs.join("tantivy.md"),
|
||||||
|
"# Tantivy BM25\nTantivy indexes rust documents.",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
fs::write(alice_docs.join("private.txt"), "alice private rust note").unwrap();
|
||||||
|
|
||||||
|
run_ok(&init_args(
|
||||||
|
&bob_config,
|
||||||
|
"bob",
|
||||||
|
bob_port,
|
||||||
|
relay_port,
|
||||||
|
&root.path().join("bob-data"),
|
||||||
|
));
|
||||||
|
run_ok(&init_args(
|
||||||
|
&alice_config,
|
||||||
|
"alice",
|
||||||
|
alice_port,
|
||||||
|
relay_port,
|
||||||
|
&root.path().join("alice-data"),
|
||||||
|
));
|
||||||
|
run_ok(&add_args(&bob_config, &bob_docs, "shared", true));
|
||||||
|
run_ok(&add_args(&alice_config, &alice_docs, "mine", false));
|
||||||
|
|
||||||
|
let bob_serve = vec![
|
||||||
|
"--config".to_string(),
|
||||||
|
bob_config.display().to_string(),
|
||||||
|
"serve".to_string(),
|
||||||
|
];
|
||||||
|
let alice_serve = vec![
|
||||||
|
"--config".to_string(),
|
||||||
|
alice_config.display().to_string(),
|
||||||
|
"serve".to_string(),
|
||||||
|
];
|
||||||
|
let _bob = spawn_service(&bob_serve, "frxd listening");
|
||||||
|
let _alice = spawn_service(&alice_serve, "frxd listening");
|
||||||
|
std::thread::sleep(Duration::from_millis(400));
|
||||||
|
|
||||||
|
let alice_config_arg = alice_config.display().to_string();
|
||||||
|
let stdout = run_ok(&[
|
||||||
|
"--config".to_string(),
|
||||||
|
alice_config_arg.clone(),
|
||||||
|
"query".to_string(),
|
||||||
|
"tantivy".to_string(),
|
||||||
|
]);
|
||||||
|
assert!(stdout.contains("Tantivy BM25"), "{stdout}");
|
||||||
|
assert!(stdout.contains("remote: 1 response(s)"), "{stdout}");
|
||||||
|
|
||||||
|
let stdout = run_ok(&[
|
||||||
|
"--config".to_string(),
|
||||||
|
alice_config_arg.clone(),
|
||||||
|
"query".to_string(),
|
||||||
|
"rust".to_string(),
|
||||||
|
"--local-only".to_string(),
|
||||||
|
]);
|
||||||
|
assert!(stdout.contains("alice private rust note"), "{stdout}");
|
||||||
|
assert!(stdout.contains("remote: 0 response(s)"), "{stdout}");
|
||||||
|
|
||||||
|
let output = frx()
|
||||||
|
.args(["--config", &alice_config_arg, "query", "tantivy"])
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
assert!(output.status.success());
|
||||||
|
assert!(String::from_utf8_lossy(&output.stdout).contains("Tantivy BM25"));
|
||||||
|
}
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
#![allow(dead_code)]
|
||||||
|
use std::path::Path;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use frxd::config::{Config, IndexSection, NodeSection, QuerySection};
|
||||||
|
use frxd::crypto::Keypair;
|
||||||
|
use frxd::index::Collection;
|
||||||
|
use frxd::message::{Envelope, QueryBody, TYPE_QUERY};
|
||||||
|
use frxd::relay;
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
pub fn config_for(dir: &Path, name: &str, relay_url: &str) -> Config {
|
||||||
|
let config = Config {
|
||||||
|
node: NodeSection {
|
||||||
|
name: name.to_string(),
|
||||||
|
listen: "127.0.0.1:0".to_string(),
|
||||||
|
relays: vec![relay_url.to_string()],
|
||||||
|
trusted_keys: Vec::new(),
|
||||||
|
responder: true,
|
||||||
|
},
|
||||||
|
query: QuerySection {
|
||||||
|
max_results: 5,
|
||||||
|
timeout_ms: 700,
|
||||||
|
},
|
||||||
|
index: IndexSection {
|
||||||
|
data_dir: dir.join("data").display().to_string(),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
config.save_key(&Keypair::generate()).unwrap();
|
||||||
|
config
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn collection(name: &str, path: &Path, shared: bool, exposure: &str) -> Collection {
|
||||||
|
Collection {
|
||||||
|
name: name.to_string(),
|
||||||
|
path: path.display().to_string(),
|
||||||
|
shared,
|
||||||
|
exposure: exposure.to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn spawn_relay() -> String {
|
||||||
|
spawn_relay_with_capacity(256).await
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn spawn_relay_with_capacity(capacity: usize) -> String {
|
||||||
|
let (listener, addr) = relay::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let _ = relay::run(listener, capacity).await;
|
||||||
|
});
|
||||||
|
format!("http://{addr}")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn free_port() -> u16 {
|
||||||
|
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
|
||||||
|
listener.local_addr().unwrap().port()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn query_envelope(key: &Keypair, text: &str, max_results: usize) -> Envelope {
|
||||||
|
let body = QueryBody::new(text, max_results);
|
||||||
|
Envelope::new(key, TYPE_QUERY, serde_json::to_value(&body).unwrap())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn register(client: &reqwest::Client, relay_url: &str, member: &str) {
|
||||||
|
poll(client, relay_url, member, 30).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn poll_messages(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
relay_url: &str,
|
||||||
|
member: &str,
|
||||||
|
timeout_ms: u64,
|
||||||
|
) -> Vec<Value> {
|
||||||
|
let response = poll(client, relay_url, member, timeout_ms).await;
|
||||||
|
if response.status() == reqwest::StatusCode::NO_CONTENT {
|
||||||
|
return Vec::new();
|
||||||
|
}
|
||||||
|
let payload: Value = response.json().await.unwrap();
|
||||||
|
payload
|
||||||
|
.get("messages")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.cloned()
|
||||||
|
.unwrap_or_default()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn messages_of_type(messages: &[Value], msg_type: &str) -> Vec<Value> {
|
||||||
|
messages
|
||||||
|
.iter()
|
||||||
|
.filter(|m| m.get("type").and_then(Value::as_str) == Some(msg_type))
|
||||||
|
.cloned()
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn client() -> reqwest::Client {
|
||||||
|
reqwest::Client::builder()
|
||||||
|
.timeout(Duration::from_secs(10))
|
||||||
|
.build()
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn ask(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
addr: &str,
|
||||||
|
text: &str,
|
||||||
|
max: usize,
|
||||||
|
) -> (reqwest::StatusCode, String, Value) {
|
||||||
|
let response = client
|
||||||
|
.post(format!("http://{addr}/v1/local/query"))
|
||||||
|
.json(&serde_json::json!({
|
||||||
|
"text": text,
|
||||||
|
"max_results": max,
|
||||||
|
"timeout_ms": 700,
|
||||||
|
}))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let status = response.status();
|
||||||
|
let raw = response.text().await.unwrap();
|
||||||
|
let value: Value = serde_json::from_str(&raw).unwrap();
|
||||||
|
(status, raw, value)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn publish(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
relay_url: &str,
|
||||||
|
envelope: &Envelope,
|
||||||
|
) -> reqwest::Response {
|
||||||
|
client
|
||||||
|
.post(format!("{relay_url}/v1/publish"))
|
||||||
|
.json(envelope)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn unicast(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
relay_url: &str,
|
||||||
|
to: &str,
|
||||||
|
envelope: &Envelope,
|
||||||
|
) -> reqwest::Response {
|
||||||
|
client
|
||||||
|
.post(format!("{relay_url}/v1/unicast?to={to}"))
|
||||||
|
.json(envelope)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn poll(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
relay_url: &str,
|
||||||
|
member: &str,
|
||||||
|
timeout_ms: u64,
|
||||||
|
) -> reqwest::Response {
|
||||||
|
client
|
||||||
|
.get(format!(
|
||||||
|
"{relay_url}/v1/poll?member={member}&timeout_ms={timeout_ms}"
|
||||||
|
))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
@@ -0,0 +1,210 @@
|
|||||||
|
mod common;
|
||||||
|
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use common::{
|
||||||
|
ask, client, collection, config_for, poll_messages, publish, query_envelope, register,
|
||||||
|
spawn_relay,
|
||||||
|
};
|
||||||
|
use frxd::crypto::Keypair;
|
||||||
|
use frxd::index::LocalIndex;
|
||||||
|
use frxd::message::{EXPOSURE_FULL, TYPE_RESPONSE};
|
||||||
|
use frxd::node::{self, Node, NodeHandle};
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
fn corpus_dir(root: &Path, name: &str, files: &[(&str, &str)]) -> std::path::PathBuf {
|
||||||
|
let dir = root.join(format!("{name}-docs"));
|
||||||
|
fs::create_dir_all(&dir).unwrap();
|
||||||
|
for (file, text) in files {
|
||||||
|
fs::write(dir.join(file), text).unwrap();
|
||||||
|
}
|
||||||
|
dir
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn start_node_with_corpus(
|
||||||
|
root: &Path,
|
||||||
|
name: &str,
|
||||||
|
relays: Vec<String>,
|
||||||
|
files: &[(&str, &str)],
|
||||||
|
) -> NodeHandle {
|
||||||
|
let docs = corpus_dir(root, name, files);
|
||||||
|
let mut config = config_for(&root.join(name), name, relays.first().unwrap());
|
||||||
|
config.node.relays = relays;
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
Node::start(config).await.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn concurrent_queries_each_get_their_response() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
vec![relay_url.clone()],
|
||||||
|
&[("doc.txt", "concurrent rust document")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
vec![relay_url.clone()],
|
||||||
|
&[("mine.txt", "alice notes")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let http = client();
|
||||||
|
let addr = alice.addr.to_string();
|
||||||
|
let mut tasks = Vec::new();
|
||||||
|
for _ in 0..4 {
|
||||||
|
let http = http.clone();
|
||||||
|
let addr = addr.clone();
|
||||||
|
tasks.push(tokio::spawn(
|
||||||
|
async move { ask(&http, &addr, "rust", 5).await },
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let mut qids = BTreeSet::new();
|
||||||
|
for task in tasks {
|
||||||
|
let (_status, _raw, value) = task.await.unwrap();
|
||||||
|
let qid = value
|
||||||
|
.get("qid")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.unwrap()
|
||||||
|
.to_string();
|
||||||
|
qids.insert(qid);
|
||||||
|
let responses = value.get("responses").and_then(Value::as_array).unwrap();
|
||||||
|
assert_eq!(responses.len(), 1, "query lost its response: {value}");
|
||||||
|
}
|
||||||
|
assert_eq!(qids.len(), 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn multi_relay_deduplicates_and_responds_once() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_one = spawn_relay().await;
|
||||||
|
let relay_two = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
vec![relay_one.clone(), relay_two.clone()],
|
||||||
|
&[("doc.txt", "dual relay rust document")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let http = client();
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
register(&http, &relay_one, &alice.public_hex()).await;
|
||||||
|
register(&http, &relay_two, &alice.public_hex()).await;
|
||||||
|
let envelope = query_envelope(&alice, "rust", 5);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_one, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_two, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
|
||||||
|
let deadline = Instant::now() + Duration::from_millis(1200);
|
||||||
|
let mut responses = Vec::new();
|
||||||
|
while responses.is_empty() && Instant::now() < deadline {
|
||||||
|
for relay in [&relay_one, &relay_two] {
|
||||||
|
let messages = poll_messages(&http, relay, &alice.public_hex(), 200).await;
|
||||||
|
responses.extend(
|
||||||
|
messages
|
||||||
|
.iter()
|
||||||
|
.filter(|m| m.get("type").and_then(Value::as_str) == Some(TYPE_RESPONSE))
|
||||||
|
.cloned(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
responses.len(),
|
||||||
|
1,
|
||||||
|
"duplicate relay delivery was answered twice"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn index_persists_across_node_restart() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let docs = corpus_dir(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&[("doc.txt", "persistent rust document")],
|
||||||
|
);
|
||||||
|
let config = config_for(&root.path().join("bob"), "bob", "http://127.0.0.1:1");
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let first = Node::start(config.clone()).await.unwrap();
|
||||||
|
let outcome = node::control_query(
|
||||||
|
&format!("http://{}", first.addr),
|
||||||
|
"rust",
|
||||||
|
Some(5),
|
||||||
|
Some(100),
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
outcome.pointer("/local/total").and_then(Value::as_u64),
|
||||||
|
Some(1)
|
||||||
|
);
|
||||||
|
drop(first);
|
||||||
|
|
||||||
|
let second = Node::start(config).await.unwrap();
|
||||||
|
let outcome = node::control_query(
|
||||||
|
&format!("http://{}", second.addr),
|
||||||
|
"rust",
|
||||||
|
Some(5),
|
||||||
|
Some(100),
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
outcome.pointer("/local/total").and_then(Value::as_u64),
|
||||||
|
Some(1)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn poll_returns_queued_batch_in_one_call() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let member = Keypair::generate();
|
||||||
|
register(&http, &relay_url, &member.public_hex()).await;
|
||||||
|
for index in 0..3 {
|
||||||
|
let envelope = query_envelope(&member, &format!("query {index}"), 5);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let messages = poll_messages(&http, &relay_url, &member.public_hex(), 300).await;
|
||||||
|
assert_eq!(messages.len(), 3);
|
||||||
|
assert!(
|
||||||
|
poll_messages(&http, &relay_url, &member.public_hex(), 50)
|
||||||
|
.await
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,924 @@
|
|||||||
|
mod common;
|
||||||
|
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use common::{
|
||||||
|
ask, client, collection, config_for, messages_of_type, poll, poll_messages, publish,
|
||||||
|
query_envelope, register, spawn_relay, spawn_relay_with_capacity, unicast,
|
||||||
|
};
|
||||||
|
use frxd::crypto::Keypair;
|
||||||
|
use frxd::index::LocalIndex;
|
||||||
|
use frxd::message::{
|
||||||
|
EXPOSURE_FULL, EXPOSURE_METADATA, Envelope, QueryBody, TYPE_AGGREGATE, TYPE_QUERY,
|
||||||
|
TYPE_RESPONSE,
|
||||||
|
};
|
||||||
|
use frxd::node::{Node, NodeHandle};
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
|
||||||
|
fn keys(value: &Value) -> BTreeSet<String> {
|
||||||
|
value.as_object().expect("object").keys().cloned().collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn status(http: &reqwest::Client, addr: &str) -> Value {
|
||||||
|
http.get(format!("http://{addr}/v1/local/status"))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn corpus_dir(root: &Path, name: &str, files: &[(&str, &str)]) -> PathBuf {
|
||||||
|
let dir = root.join(format!("{name}-docs"));
|
||||||
|
fs::create_dir_all(&dir).unwrap();
|
||||||
|
for (file, text) in files {
|
||||||
|
fs::write(dir.join(file), text).unwrap();
|
||||||
|
}
|
||||||
|
dir
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn start_node_with_corpus(
|
||||||
|
root: &Path,
|
||||||
|
name: &str,
|
||||||
|
relay_url: &str,
|
||||||
|
shared: bool,
|
||||||
|
exposure: &str,
|
||||||
|
files: &[(&str, &str)],
|
||||||
|
) -> NodeHandle {
|
||||||
|
let docs = corpus_dir(root, name, files);
|
||||||
|
let config = config_for(&root.join(name), name, relay_url);
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("docs", &docs, shared, exposure))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
Node::start(config).await.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn collect_responses(
|
||||||
|
http: &reqwest::Client,
|
||||||
|
relay_url: &str,
|
||||||
|
member: &str,
|
||||||
|
timeout_ms: u64,
|
||||||
|
) -> Vec<Value> {
|
||||||
|
let start = Instant::now();
|
||||||
|
let deadline = Duration::from_millis(timeout_ms);
|
||||||
|
let mut responses = Vec::new();
|
||||||
|
while start.elapsed() < deadline {
|
||||||
|
let messages = poll_messages(http, relay_url, member, 200).await;
|
||||||
|
responses.extend(messages_of_type(&messages, TYPE_RESPONSE));
|
||||||
|
if !responses.is_empty() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
responses
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn raw_query(
|
||||||
|
http: &reqwest::Client,
|
||||||
|
relay_url: &str,
|
||||||
|
asker: &Keypair,
|
||||||
|
text: &str,
|
||||||
|
timeout_ms: u64,
|
||||||
|
) -> Vec<Value> {
|
||||||
|
register(http, relay_url, &asker.public_hex()).await;
|
||||||
|
let envelope = query_envelope(asker, text, 5);
|
||||||
|
assert!(
|
||||||
|
publish(http, relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
collect_responses(http, relay_url, &asker.public_hex(), timeout_ms).await
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn envelope_field_set_is_fixed() {
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let envelope = Envelope::new(&key, TYPE_QUERY, json!({"qid": "q", "text": "t"}));
|
||||||
|
let value = serde_json::to_value(&envelope).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
keys(&value),
|
||||||
|
BTreeSet::from([
|
||||||
|
"body".to_string(),
|
||||||
|
"from".to_string(),
|
||||||
|
"nonce".to_string(),
|
||||||
|
"sig".to_string(),
|
||||||
|
"ts".to_string(),
|
||||||
|
"type".to_string(),
|
||||||
|
])
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn ordering_travels_scores_dont() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[
|
||||||
|
("strong.txt", "rust rust rust rust search engine"),
|
||||||
|
("weak.txt", "rust notes"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
let responses = raw_query(&http, &relay_url, &alice, "rust", 700).await;
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
let results = responses[0]
|
||||||
|
.pointer("/body/results")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(results.len(), 2);
|
||||||
|
let first_url = results[0].get("url").and_then(Value::as_str).unwrap();
|
||||||
|
assert!(
|
||||||
|
first_url.contains("strong"),
|
||||||
|
"best match should lead the ordering: {first_url}"
|
||||||
|
);
|
||||||
|
for result in results {
|
||||||
|
assert!(!keys(result).contains("score"));
|
||||||
|
assert!(!keys(result).contains("rank"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn metadata_exposure_withholds_content() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_METADATA,
|
||||||
|
&[("corpus.txt", "enrichment corpus about rust")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
let responses = raw_query(&http, &relay_url, &alice, "rust", 700).await;
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
let result = &responses[0]
|
||||||
|
.pointer("/body/results")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()[0];
|
||||||
|
assert_eq!(
|
||||||
|
result.get("exposure").and_then(Value::as_str),
|
||||||
|
Some(EXPOSURE_METADATA)
|
||||||
|
);
|
||||||
|
assert!(result.get("content").unwrap().is_null());
|
||||||
|
assert!(
|
||||||
|
!result
|
||||||
|
.get("summary")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.unwrap()
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn full_exposure_carries_content() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("corpus.txt", "full exposure rust content body")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
let responses = raw_query(&http, &relay_url, &alice, "rust", 700).await;
|
||||||
|
let result = &responses[0]
|
||||||
|
.pointer("/body/results")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()[0];
|
||||||
|
assert_eq!(
|
||||||
|
result.get("exposure").and_then(Value::as_str),
|
||||||
|
Some(EXPOSURE_FULL)
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
result
|
||||||
|
.get("content")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.unwrap()
|
||||||
|
.contains("full exposure rust content body")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn private_collections_are_silent_egress() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
false,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("private.txt", "rust private data")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
let responses = raw_query(&http, &relay_url, &alice, "rust", 500).await;
|
||||||
|
assert!(responses.is_empty(), "private collection answered");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn one_result_is_a_conformant_response() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("only.txt", "solitary rust document")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
let responses = raw_query(&http, &relay_url, &alice, "solitary", 700).await;
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0]
|
||||||
|
.pointer("/body/results")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0]
|
||||||
|
.pointer("/body/truncated")
|
||||||
|
.and_then(Value::as_bool),
|
||||||
|
Some(false)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn empty_query_earns_silence() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "rust content")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
&relay_url,
|
||||||
|
false,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("mine.txt", "alice rust")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let response = client()
|
||||||
|
.post(format!("http://{}/v1/local/query", alice.addr))
|
||||||
|
.json(&json!({"text": "", "max_results": 5, "timeout_ms": 500}))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json::<Value>()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
response
|
||||||
|
.get("responses")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
assert!(bob.node.doc_count() > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn entities_are_optional_hints() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "entity tagged rust document")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
register(&http, &relay_url, &alice.public_hex()).await;
|
||||||
|
let body = json!({
|
||||||
|
"qid": "entities-test",
|
||||||
|
"text": "rust",
|
||||||
|
"entities": ["Q999999", "not-a-real-entity"],
|
||||||
|
"budget": {"max_results": 5}
|
||||||
|
});
|
||||||
|
let envelope = Envelope::new(&alice, TYPE_QUERY, body);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
let responses = collect_responses(&http, &relay_url, &alice.public_hex(), 700).await;
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0].pointer("/body/qid").and_then(Value::as_str),
|
||||||
|
Some("entities-test")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn trusted_keys_allowlist_filters_senders() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let untrusted = Keypair::generate();
|
||||||
|
|
||||||
|
let docs = corpus_dir(root.path(), "bob", &[("doc.txt", "rust document")]);
|
||||||
|
let mut config = config_for(&root.path().join("bob"), "bob", &relay_url);
|
||||||
|
config.node.trusted_keys = vec![alice.public_hex()];
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("docs", &docs, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
let _bob = Node::start(config).await.unwrap();
|
||||||
|
|
||||||
|
let http = client();
|
||||||
|
let trusted = raw_query(&http, &relay_url, &alice, "rust", 700).await;
|
||||||
|
assert_eq!(trusted.len(), 1, "trusted member got no response");
|
||||||
|
|
||||||
|
register(&http, &relay_url, &untrusted.public_hex()).await;
|
||||||
|
let envelope = query_envelope(&untrusted, "rust", 5);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
let denied = collect_responses(&http, &relay_url, &untrusted.public_hex(), 400).await;
|
||||||
|
assert!(denied.is_empty(), "untrusted member received a response");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn backpressure_is_visible_and_recoverable() {
|
||||||
|
let relay_url = spawn_relay_with_capacity(1).await;
|
||||||
|
let http = client();
|
||||||
|
let member = Keypair::generate();
|
||||||
|
let publisher = Keypair::generate();
|
||||||
|
register(&http, &relay_url, &member.public_hex()).await;
|
||||||
|
|
||||||
|
let first = query_envelope(&publisher, "one", 5);
|
||||||
|
assert_eq!(
|
||||||
|
publish(&http, &relay_url, &first).await.status(),
|
||||||
|
reqwest::StatusCode::ACCEPTED
|
||||||
|
);
|
||||||
|
let second = query_envelope(&publisher, "two", 5);
|
||||||
|
let response = publish(&http, &relay_url, &second).await;
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::TOO_MANY_REQUESTS);
|
||||||
|
assert_eq!(
|
||||||
|
response
|
||||||
|
.headers()
|
||||||
|
.get("retry-after")
|
||||||
|
.and_then(|v| v.to_str().ok()),
|
||||||
|
Some("1")
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
poll_messages(&http, &relay_url, &member.public_hex(), 300)
|
||||||
|
.await
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
poll_messages(&http, &relay_url, &publisher.public_hex(), 300)
|
||||||
|
.await
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
|
||||||
|
let third = query_envelope(&publisher, "three", 5);
|
||||||
|
assert_eq!(
|
||||||
|
publish(&http, &relay_url, &third).await.status(),
|
||||||
|
reqwest::StatusCode::ACCEPTED
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn unicast_is_need_to_know() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let bob = Keypair::generate();
|
||||||
|
register(&http, &relay_url, &alice.public_hex()).await;
|
||||||
|
register(&http, &relay_url, &bob.public_hex()).await;
|
||||||
|
|
||||||
|
let query = query_envelope(&alice, "rust", 5);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &query)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
|
||||||
|
let response = Envelope::new(
|
||||||
|
&bob,
|
||||||
|
TYPE_RESPONSE,
|
||||||
|
json!({"qid": "q1", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
unicast(&http, &relay_url, &alice.public_hex(), &response)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::OK
|
||||||
|
);
|
||||||
|
|
||||||
|
let alice_messages = poll_messages(&http, &relay_url, &alice.public_hex(), 300).await;
|
||||||
|
assert_eq!(messages_of_type(&alice_messages, TYPE_RESPONSE).len(), 1);
|
||||||
|
let bob_messages = poll_messages(&http, &relay_url, &bob.public_hex(), 300).await;
|
||||||
|
assert!(
|
||||||
|
messages_of_type(&bob_messages, TYPE_RESPONSE).is_empty(),
|
||||||
|
"unicast response leaked to another member"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn unicast_to_unknown_member_is_visible() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let sender = Keypair::generate();
|
||||||
|
let response = Envelope::new(
|
||||||
|
&sender,
|
||||||
|
TYPE_RESPONSE,
|
||||||
|
json!({"qid": "q1", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
|
||||||
|
);
|
||||||
|
let stranger = Keypair::generate().public_hex();
|
||||||
|
assert_eq!(
|
||||||
|
unicast(&http, &relay_url, &stranger, &response)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::NOT_FOUND
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn tampered_envelopes_are_rejected() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let mut envelope = query_envelope(&key, "rust", 5);
|
||||||
|
envelope.body = json!({"qid": "q1", "text": "forged", "budget": {"max_results": 5}});
|
||||||
|
let publish_response = publish(&http, &relay_url, &envelope).await;
|
||||||
|
assert_eq!(publish_response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||||
|
|
||||||
|
let mut response = Envelope::new(
|
||||||
|
&key,
|
||||||
|
TYPE_RESPONSE,
|
||||||
|
json!({"qid": "q1", "results": [], "truncated": false, "more_available": 0, "cursor": null}),
|
||||||
|
);
|
||||||
|
response.sig = "00".repeat(64);
|
||||||
|
let unicast_response = unicast(&http, &relay_url, &key.public_hex(), &response).await;
|
||||||
|
assert_eq!(unicast_response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn invalid_poll_member_is_rejected() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let response = poll(&client(), &relay_url, "not-hex-at-all", 30).await;
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn malformed_json_is_rejected() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let response = http
|
||||||
|
.post(format!("{relay_url}/v1/publish"))
|
||||||
|
.header("content-type", "application/json")
|
||||||
|
.body("{not json")
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||||
|
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let config = config_for(&root.path().join("alice"), "alice", &relay_url);
|
||||||
|
let node = Node::start(config).await.unwrap();
|
||||||
|
let response = http
|
||||||
|
.post(format!("http://{}/v1/local/query", node.addr))
|
||||||
|
.header("content-type", "application/json")
|
||||||
|
.body("{not json")
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn node_does_not_answer_its_own_queries() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "self query rust")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let (_status, _raw, value) = ask(&client(), &bob.addr.to_string(), "rust", 5).await;
|
||||||
|
assert!(
|
||||||
|
value
|
||||||
|
.get("responses")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.is_empty(),
|
||||||
|
"node responded to itself: {value}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn ingress_happens_only_locally() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "rust served by bob")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
&relay_url,
|
||||||
|
false,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("mine.txt", "alice rust notes")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let http = client();
|
||||||
|
let bob_before = status(&http, &bob.addr.to_string()).await;
|
||||||
|
let alice_before = status(&http, &alice.addr.to_string()).await;
|
||||||
|
|
||||||
|
let (_s, _r, value) = ask(&http, &alice.addr.to_string(), "rust", 5).await;
|
||||||
|
assert_eq!(
|
||||||
|
value
|
||||||
|
.get("responses")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
|
||||||
|
let bob_after = status(&http, &bob.addr.to_string()).await;
|
||||||
|
let alice_after = status(&http, &alice.addr.to_string()).await;
|
||||||
|
assert_eq!(bob_before.get("doc_count"), bob_after.get("doc_count"));
|
||||||
|
assert_eq!(alice_before.get("doc_count"), alice_after.get("doc_count"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn querier_and_responder_roles_coexist() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("a.txt", "alice rust document")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("b.txt", "bob rust document")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let http = client();
|
||||||
|
let alice_addr = alice.addr.to_string();
|
||||||
|
let bob_addr = bob.addr.to_string();
|
||||||
|
let alice_ask = async { common::ask(&http, &alice_addr, "rust", 5).await };
|
||||||
|
let bob_ask = async { common::ask(&http, &bob_addr, "rust", 5).await };
|
||||||
|
let (alice_result, bob_result) = tokio::join!(alice_ask, bob_ask);
|
||||||
|
|
||||||
|
for (name, (_status, _raw, value)) in [("alice", alice_result), ("bob", bob_result)] {
|
||||||
|
let responses = value.get("responses").and_then(Value::as_array).unwrap();
|
||||||
|
assert_eq!(responses.len(), 1, "{name} got {value}");
|
||||||
|
let member = responses[0].get("member").and_then(Value::as_str).unwrap();
|
||||||
|
assert!(!member.is_empty());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn budget_is_clamped_and_never_exceeded() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[
|
||||||
|
("one.txt", "rust one"),
|
||||||
|
("two.txt", "rust two"),
|
||||||
|
("three.txt", "rust three"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
&relay_url,
|
||||||
|
false,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("mine.txt", "alice doc")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let http = client();
|
||||||
|
|
||||||
|
let (_s, _r, value) = ask(&http, &alice.addr.to_string(), "rust", 0).await;
|
||||||
|
let responses = value.get("responses").and_then(Value::as_array).unwrap();
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0]
|
||||||
|
.get("results")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
1,
|
||||||
|
"zero budget must clamp to one, never send nothing or more"
|
||||||
|
);
|
||||||
|
|
||||||
|
let (_s, _r, value) = ask(&http, &alice.addr.to_string(), "rust", 1000).await;
|
||||||
|
let responses = value.get("responses").and_then(Value::as_array).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
responses[0]
|
||||||
|
.get("results")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
3
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0].get("truncated").and_then(Value::as_bool),
|
||||||
|
Some(false)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn aggregates_are_unicast_only_and_ignored_by_nodes() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "aggregate test rust")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let http = client();
|
||||||
|
let sender = Keypair::generate();
|
||||||
|
register(&http, &relay_url, &bob.pubkey).await;
|
||||||
|
|
||||||
|
let broadcast = Envelope::new(
|
||||||
|
&sender,
|
||||||
|
TYPE_AGGREGATE,
|
||||||
|
json!({"period": "2026-03", "sent": 1, "passed": 0, "cited": 0}),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
publish(&http, &relay_url, &broadcast).await.status(),
|
||||||
|
reqwest::StatusCode::BAD_REQUEST,
|
||||||
|
"aggregates must not travel the broadcast channel"
|
||||||
|
);
|
||||||
|
|
||||||
|
let unicast_envelope = Envelope::new(
|
||||||
|
&sender,
|
||||||
|
TYPE_AGGREGATE,
|
||||||
|
json!({"period": "2026-03", "sent": 1, "passed": 0, "cited": 0}),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
unicast(&http, &relay_url, &bob.pubkey, &unicast_envelope)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::OK
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||||
|
let (_s, _r, value) = ask(&http, &bob.addr.to_string(), "ignored", 5).await;
|
||||||
|
assert_eq!(
|
||||||
|
value.pointer("/local/total").and_then(Value::as_u64),
|
||||||
|
Some(0)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn node_survives_unreachable_relay() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
"http://127.0.0.1:1",
|
||||||
|
false,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("mine.txt", "offline rust note")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let (status_code, _raw, value) = ask(&client(), &alice.addr.to_string(), "rust", 5).await;
|
||||||
|
assert!(status_code.is_success());
|
||||||
|
assert_eq!(
|
||||||
|
value.pointer("/local/total").and_then(Value::as_u64),
|
||||||
|
Some(1)
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
value
|
||||||
|
.get("responses")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.unwrap()
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn duplicate_delivery_is_answered_once() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "duplicate rust doc")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
register(&http, &relay_url, &alice.public_hex()).await;
|
||||||
|
let envelope = query_envelope(&alice, "rust", 5);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
let responses = collect_responses(&http, &relay_url, &alice.public_hex(), 700).await;
|
||||||
|
assert_eq!(
|
||||||
|
responses.len(),
|
||||||
|
1,
|
||||||
|
"duplicate query produced duplicate responses"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn multiple_responders_merge_with_provenance() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("bob.txt", "bob rust contribution")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let _carol = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"carol",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("carol.txt", "carol rust contribution")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"alice",
|
||||||
|
&relay_url,
|
||||||
|
false,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("mine.txt", "alice rust local")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let (_s, _r, value) = ask(&client(), &alice.addr.to_string(), "rust", 5).await;
|
||||||
|
let responses = value.get("responses").and_then(Value::as_array).unwrap();
|
||||||
|
assert_eq!(responses.len(), 2, "expected both responders: {value}");
|
||||||
|
let members: BTreeSet<&str> = responses
|
||||||
|
.iter()
|
||||||
|
.map(|r| r.get("member").and_then(Value::as_str).unwrap())
|
||||||
|
.collect();
|
||||||
|
assert_eq!(members.len(), 2);
|
||||||
|
let merged = value.get("merged").and_then(Value::as_array).unwrap();
|
||||||
|
let provenances: BTreeSet<&str> = merged
|
||||||
|
.iter()
|
||||||
|
.map(|m| m.get("provenance").and_then(Value::as_str).unwrap())
|
||||||
|
.collect();
|
||||||
|
assert!(provenances.contains("local"));
|
||||||
|
assert_eq!(provenances.len(), 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn canonical_signing_bytes_are_stable() {
|
||||||
|
let mut envelope = Envelope {
|
||||||
|
msg_type: TYPE_QUERY.to_string(),
|
||||||
|
from: "aa".repeat(32),
|
||||||
|
ts: 1_700_000_000,
|
||||||
|
nonce: "00112233445566778899aabbccddeeff".to_string(),
|
||||||
|
body: json!({"text": "rust", "qid": "q1", "budget": {"max_results": 5}, "entities": []}),
|
||||||
|
sig: String::new(),
|
||||||
|
};
|
||||||
|
let first = frxd::crypto::signing_bytes(&envelope);
|
||||||
|
envelope.body =
|
||||||
|
json!({"entities": [], "budget": {"max_results": 5}, "qid": "q1", "text": "rust"});
|
||||||
|
let second = frxd::crypto::signing_bytes(&envelope);
|
||||||
|
assert_eq!(first, second);
|
||||||
|
assert_eq!(
|
||||||
|
String::from_utf8(first).unwrap(),
|
||||||
|
"FRX/0.3\nquery\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n1700000000\n00112233445566778899aabbccddeeff\n{\"budget\":{\"max_results\":5},\"entities\":[],\"qid\":\"q1\",\"text\":\"rust\"}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn query_body_without_entities_parses() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let _bob = start_node_with_corpus(
|
||||||
|
root.path(),
|
||||||
|
"bob",
|
||||||
|
&relay_url,
|
||||||
|
true,
|
||||||
|
EXPOSURE_FULL,
|
||||||
|
&[("doc.txt", "minimal body rust")],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let http = client();
|
||||||
|
register(&http, &relay_url, &alice.public_hex()).await;
|
||||||
|
let envelope = Envelope::new(
|
||||||
|
&alice,
|
||||||
|
TYPE_QUERY,
|
||||||
|
json!({"qid": "min", "text": "rust", "budget": {"max_results": 5}}),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
let responses = collect_responses(&http, &relay_url, &alice.public_hex(), 700).await;
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn query_body_budget_is_the_only_quantity_knob() {
|
||||||
|
let value = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
keys(&value),
|
||||||
|
BTreeSet::from([
|
||||||
|
"qid".to_string(),
|
||||||
|
"text".to_string(),
|
||||||
|
"entities".to_string(),
|
||||||
|
"budget".to_string()
|
||||||
|
])
|
||||||
|
);
|
||||||
|
}
|
||||||
+116
@@ -0,0 +1,116 @@
|
|||||||
|
mod common;
|
||||||
|
|
||||||
|
use std::fs;
|
||||||
|
|
||||||
|
use common::{ask, client, collection, config_for, publish, spawn_relay};
|
||||||
|
use frxd::index::LocalIndex;
|
||||||
|
use frxd::message::{EXPOSURE_FULL, Envelope, TYPE_RESPONSE};
|
||||||
|
use frxd::node::Node;
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn two_node_broadcast_query_flow() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
|
||||||
|
let shared_dir = root.path().join("bob/shared");
|
||||||
|
let private_dir = root.path().join("bob/private");
|
||||||
|
fs::create_dir_all(&shared_dir).unwrap();
|
||||||
|
fs::create_dir_all(&private_dir).unwrap();
|
||||||
|
fs::write(
|
||||||
|
shared_dir.join("alpha.txt"),
|
||||||
|
"Tantivy provides BM25 relevance scoring for rust search",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
fs::write(
|
||||||
|
shared_dir.join("beta.txt"),
|
||||||
|
"Rust lifetimes and borrowing explained",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
fs::write(private_dir.join("gamma.txt"), "rust secret launch codes").unwrap();
|
||||||
|
|
||||||
|
let bob_config = config_for(&root.path().join("bob"), "bob", &relay_url);
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&bob_config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("shared", &shared_dir, true, EXPOSURE_FULL))
|
||||||
|
.unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("private", &private_dir, false, "metadata"))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
let _bob = Node::start(bob_config).await.unwrap();
|
||||||
|
|
||||||
|
let alice_dir = root.path().join("alice");
|
||||||
|
let alice_local = root.path().join("alice_mine");
|
||||||
|
fs::create_dir_all(&alice_local).unwrap();
|
||||||
|
fs::write(alice_local.join("notes.txt"), "my rust notes").unwrap();
|
||||||
|
let alice_config = config_for(&alice_dir, "alice", &relay_url);
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&alice_config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("mine", &alice_local, false, "metadata"))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
let alice = Node::start(alice_config).await.unwrap();
|
||||||
|
|
||||||
|
let http = client();
|
||||||
|
let addr = alice.addr.to_string();
|
||||||
|
|
||||||
|
let (status, raw, value) = ask(&http, &addr, "rust", 1).await;
|
||||||
|
assert!(status.is_success());
|
||||||
|
assert_eq!(
|
||||||
|
value.pointer("/local/total").and_then(|v| v.as_u64()),
|
||||||
|
Some(1)
|
||||||
|
);
|
||||||
|
let responses = value.get("responses").and_then(|v| v.as_array()).unwrap();
|
||||||
|
assert_eq!(responses.len(), 1, "expected one responder: {value}");
|
||||||
|
let response = &responses[0];
|
||||||
|
let results = response.get("results").and_then(|v| v.as_array()).unwrap();
|
||||||
|
assert_eq!(results.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
response.get("truncated").and_then(|v| v.as_bool()),
|
||||||
|
Some(true)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
response.get("more_available").and_then(|v| v.as_u64()),
|
||||||
|
Some(1)
|
||||||
|
);
|
||||||
|
assert!(!raw.contains("gamma"), "unshared collection leaked: {raw}");
|
||||||
|
assert!(!raw.contains("\"score\""), "response carries scores: {raw}");
|
||||||
|
|
||||||
|
let (_status, _raw, value) = ask(&http, &addr, "rust", 5).await;
|
||||||
|
let responses = value.get("responses").and_then(|v| v.as_array()).unwrap();
|
||||||
|
assert_eq!(responses.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0]
|
||||||
|
.get("results")
|
||||||
|
.and_then(|v| v.as_array())
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
2
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
responses[0].get("truncated").and_then(|v| v.as_bool()),
|
||||||
|
Some(false)
|
||||||
|
);
|
||||||
|
|
||||||
|
let (_status, _raw, value) = ask(&http, &addr, "secret launch codes", 5).await;
|
||||||
|
let responses = value.get("responses").and_then(|v| v.as_array()).unwrap();
|
||||||
|
assert!(
|
||||||
|
responses.is_empty(),
|
||||||
|
"private collection was served: {value}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn relay_rejects_non_query_broadcasts() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let key = frxd::crypto::Keypair::generate();
|
||||||
|
let envelope = Envelope::new(
|
||||||
|
&key,
|
||||||
|
TYPE_RESPONSE,
|
||||||
|
serde_json::json!({"qid": "x", "results": []}),
|
||||||
|
);
|
||||||
|
let response = publish(&client(), &relay_url, &envelope).await;
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||||
|
}
|
||||||
+389
@@ -0,0 +1,389 @@
|
|||||||
|
mod common;
|
||||||
|
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
use std::fs;
|
||||||
|
|
||||||
|
use common::{client, collection, config_for, poll, publish, spawn_relay, unicast};
|
||||||
|
use frxd::crypto::Keypair;
|
||||||
|
use frxd::index::LocalIndex;
|
||||||
|
use frxd::message::{
|
||||||
|
AggregateBody, Envelope, QueryBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY, TYPE_RESPONSE,
|
||||||
|
build_response,
|
||||||
|
};
|
||||||
|
use frxd::node::{self, Node};
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
fn assert_exact_keys(value: &Value, expected: &[&str]) {
|
||||||
|
let actual: BTreeSet<&str> = value
|
||||||
|
.as_object()
|
||||||
|
.expect("object")
|
||||||
|
.keys()
|
||||||
|
.map(String::as_str)
|
||||||
|
.collect();
|
||||||
|
let expected: BTreeSet<&str> = expected.iter().copied().collect();
|
||||||
|
assert_eq!(actual, expected, "fields drifted: {value}");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn assert_absent_fields(value: &Value, banned: &[&str]) {
|
||||||
|
let raw = serde_json::to_string(value).unwrap();
|
||||||
|
for field in banned {
|
||||||
|
assert!(
|
||||||
|
!raw.contains(&format!("\"{field}\"")),
|
||||||
|
"purged field '{field}' present: {raw}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_bounty_winner_selection_or_slashing() {
|
||||||
|
let query = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
|
||||||
|
assert_exact_keys(&query, &["qid", "text", "entities", "budget"]);
|
||||||
|
assert_exact_keys(query.get("budget").unwrap(), &["max_results"]);
|
||||||
|
let envelope = Envelope::new(&Keypair::generate(), TYPE_QUERY, query.clone());
|
||||||
|
let envelope = serde_json::to_value(&envelope).unwrap();
|
||||||
|
for value in [&query, &envelope] {
|
||||||
|
assert_absent_fields(
|
||||||
|
value,
|
||||||
|
&[
|
||||||
|
"bounty",
|
||||||
|
"winner",
|
||||||
|
"winner_selection",
|
||||||
|
"slashing",
|
||||||
|
"stake",
|
||||||
|
"reward",
|
||||||
|
"payment",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_protocol_query_dedup() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let envelope = Envelope::new(
|
||||||
|
&key,
|
||||||
|
TYPE_QUERY,
|
||||||
|
serde_json::json!({"qid": "q1", "text": "rust"}),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
let response = poll(&http, &relay_url, &key.public_hex(), 300).await;
|
||||||
|
let payload: Value = response.json().await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
payload
|
||||||
|
.get("messages")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.map(Vec::len),
|
||||||
|
Some(2),
|
||||||
|
"relay deduplicated identical queries: {payload}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_k_fetch_ingestion_attestations() {
|
||||||
|
let response = build_response("q1", Vec::new(), 0, 5);
|
||||||
|
let value = serde_json::to_value(&response).unwrap();
|
||||||
|
assert_exact_keys(
|
||||||
|
&value,
|
||||||
|
&["qid", "results", "truncated", "more_available", "cursor"],
|
||||||
|
);
|
||||||
|
assert_absent_fields(
|
||||||
|
&value,
|
||||||
|
&[
|
||||||
|
"attestation",
|
||||||
|
"attestations",
|
||||||
|
"k_fetch",
|
||||||
|
"fetch_proof",
|
||||||
|
"ingestion_proof",
|
||||||
|
"fetch_count",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_result_count_etiquette() {
|
||||||
|
let query = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
|
||||||
|
assert_absent_fields(
|
||||||
|
&query,
|
||||||
|
&["min_results", "results_count", "serp", "count_floor"],
|
||||||
|
);
|
||||||
|
let response = serde_json::to_value(build_response("q1", Vec::new(), 0, 5)).unwrap();
|
||||||
|
assert_absent_fields(
|
||||||
|
&response,
|
||||||
|
&["min_results", "results_count", "serp", "count_floor"],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_global_reputation_score() {
|
||||||
|
let item = ResponseItem {
|
||||||
|
url: "file:///x".to_string(),
|
||||||
|
title: "t".to_string(),
|
||||||
|
summary: String::new(),
|
||||||
|
published: String::new(),
|
||||||
|
exposure: "metadata".to_string(),
|
||||||
|
content: None,
|
||||||
|
};
|
||||||
|
let item_value = serde_json::to_value(&item).unwrap();
|
||||||
|
assert_exact_keys(
|
||||||
|
&item_value,
|
||||||
|
&[
|
||||||
|
"url",
|
||||||
|
"title",
|
||||||
|
"summary",
|
||||||
|
"published",
|
||||||
|
"exposure",
|
||||||
|
"content",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
let response_value = serde_json::to_value(build_response("q1", vec![item], 1, 5)).unwrap();
|
||||||
|
for value in [&item_value, &response_value] {
|
||||||
|
assert_absent_fields(value, &["score", "rank", "reputation", "weight", "rating"]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_aggregate_appeals() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let sender = Keypair::generate();
|
||||||
|
let member = Keypair::generate();
|
||||||
|
|
||||||
|
let dispute = Envelope::new(
|
||||||
|
&sender,
|
||||||
|
"dispute",
|
||||||
|
serde_json::json!({"about": "counter", "reason": "unfair"}),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
publish(&http, &relay_url, &dispute).await.status(),
|
||||||
|
reqwest::StatusCode::BAD_REQUEST
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
unicast(&http, &relay_url, &member.public_hex(), &dispute)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::BAD_REQUEST
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
poll(&http, &relay_url, &member.public_hex(), 30)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::NO_CONTENT
|
||||||
|
);
|
||||||
|
let aggregate = Envelope::new(
|
||||||
|
&sender,
|
||||||
|
TYPE_AGGREGATE,
|
||||||
|
serde_json::to_value(AggregateBody {
|
||||||
|
period: "2026-03".to_string(),
|
||||||
|
sent: 1,
|
||||||
|
passed: 1,
|
||||||
|
cited: 0,
|
||||||
|
})
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
unicast(&http, &relay_url, &member.public_hex(), &aggregate)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::OK
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = serde_json::to_value(AggregateBody {
|
||||||
|
period: "2026-03".to_string(),
|
||||||
|
sent: 1,
|
||||||
|
passed: 1,
|
||||||
|
cited: 0,
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
assert_exact_keys(&body, &["period", "sent", "passed", "cited"]);
|
||||||
|
assert_absent_fields(&body, &["appeal", "dispute", "complaint", "sanction"]);
|
||||||
|
|
||||||
|
for route in ["/v1/appeal", "/v1/dispute"] {
|
||||||
|
let response = http
|
||||||
|
.get(format!("{relay_url}{route}"))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::NOT_FOUND);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_topic_channels() {
|
||||||
|
let query = serde_json::to_value(QueryBody::new("rust", 5)).unwrap();
|
||||||
|
assert_exact_keys(&query, &["qid", "text", "entities", "budget"]);
|
||||||
|
assert_absent_fields(
|
||||||
|
&query,
|
||||||
|
&[
|
||||||
|
"topic", "channel", "taxonomy", "category", "routing", "topic_id",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let alice = Keypair::generate();
|
||||||
|
let bob = Keypair::generate();
|
||||||
|
assert_eq!(
|
||||||
|
poll(&http, &relay_url, &alice.public_hex(), 30)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::NO_CONTENT
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
poll(&http, &relay_url, &bob.public_hex(), 30)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::NO_CONTENT
|
||||||
|
);
|
||||||
|
let envelope = Envelope::new(
|
||||||
|
&alice,
|
||||||
|
TYPE_QUERY,
|
||||||
|
serde_json::json!({"qid": "q1", "text": "rust"}),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
for member in [&alice, &bob] {
|
||||||
|
let response = poll(&http, &relay_url, &member.public_hex(), 300).await;
|
||||||
|
let payload: Value = response.json().await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
payload
|
||||||
|
.get("messages")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.map(Vec::len),
|
||||||
|
Some(1),
|
||||||
|
"receiver-side filtering expected, got {payload}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_broadcast_responses() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let key = Keypair::generate();
|
||||||
|
for msg_type in [TYPE_RESPONSE, "evidence", "commons", "broadcast_response"] {
|
||||||
|
let envelope = Envelope::new(&key, msg_type, serde_json::json!({}));
|
||||||
|
assert_eq!(
|
||||||
|
publish(&http, &relay_url, &envelope).await.status(),
|
||||||
|
reqwest::StatusCode::BAD_REQUEST,
|
||||||
|
"relay accepted broadcast {msg_type}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_normative_query_canonical_form() {
|
||||||
|
let root = tempfile::tempdir().unwrap();
|
||||||
|
let docs = root.path().join("docs");
|
||||||
|
fs::create_dir_all(&docs).unwrap();
|
||||||
|
fs::write(docs.join("alpha.txt"), "rust ownership and borrowing").unwrap();
|
||||||
|
let config = config_for(&root.path().join("alice"), "alice", "http://127.0.0.1:1");
|
||||||
|
{
|
||||||
|
let index = LocalIndex::open(&config.index_dir()).unwrap();
|
||||||
|
index
|
||||||
|
.add_collection(&collection("docs", &docs, true, "full"))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
let node = Node::start(config).await.unwrap();
|
||||||
|
let text = "rust?! ownership/borrowing: \"quotes\" ünïcode {braces} #tag";
|
||||||
|
let outcome = node::control_query(
|
||||||
|
&format!("http://{}", node.addr),
|
||||||
|
text,
|
||||||
|
Some(5),
|
||||||
|
Some(100),
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(outcome.get("text").and_then(Value::as_str), Some(text));
|
||||||
|
assert_eq!(
|
||||||
|
outcome.pointer("/local/total").and_then(Value::as_u64),
|
||||||
|
Some(1),
|
||||||
|
"arbitrary phrasing should match lexically: {outcome}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_supply_announce_firehose() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let key = Keypair::generate();
|
||||||
|
for msg_type in ["announce", "supply", "documents", "publish"] {
|
||||||
|
let envelope = Envelope::new(&key, msg_type, serde_json::json!({}));
|
||||||
|
assert_eq!(
|
||||||
|
publish(&http, &relay_url, &envelope).await.status(),
|
||||||
|
reqwest::StatusCode::BAD_REQUEST,
|
||||||
|
"relay accepted supply announcement {msg_type}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for route in ["/v1/announce", "/v1/supply"] {
|
||||||
|
let response = http
|
||||||
|
.get(format!("{relay_url}{route}"))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::NOT_FOUND);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn no_durable_replayable_broadcast_stream() {
|
||||||
|
let relay_url = spawn_relay().await;
|
||||||
|
let http = client();
|
||||||
|
let key = Keypair::generate();
|
||||||
|
let envelope = Envelope::new(
|
||||||
|
&key,
|
||||||
|
TYPE_QUERY,
|
||||||
|
serde_json::json!({"qid": "q1", "text": "rust"}),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
publish(&http, &relay_url, &envelope)
|
||||||
|
.await
|
||||||
|
.status()
|
||||||
|
.is_success()
|
||||||
|
);
|
||||||
|
|
||||||
|
let response = poll(&http, &relay_url, &key.public_hex(), 300).await;
|
||||||
|
let payload: Value = response.json().await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
payload
|
||||||
|
.get("messages")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.map(Vec::len),
|
||||||
|
Some(1)
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
poll(&http, &relay_url, &key.public_hex(), 50)
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
reqwest::StatusCode::NO_CONTENT,
|
||||||
|
"relay replayed a drained message"
|
||||||
|
);
|
||||||
|
|
||||||
|
for route in ["/v1/history", "/v1/replay"] {
|
||||||
|
let response = http
|
||||||
|
.get(format!("{relay_url}{route}"))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(response.status(), reqwest::StatusCode::NOT_FOUND);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
use std::fs;
|
||||||
|
use std::time::Instant;
|
||||||
|
|
||||||
|
use frxd::index::{Collection, LocalIndex, response_items};
|
||||||
|
use frxd::message::{EXPOSURE_FULL, build_response};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn thousand_file_corpus_is_searchable_and_honest() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let docs = temp.path().join("corpus");
|
||||||
|
fs::create_dir_all(&docs).unwrap();
|
||||||
|
let total = 1000;
|
||||||
|
for i in 0..total {
|
||||||
|
let part = docs.join(format!("part{:02}", i / 100));
|
||||||
|
fs::create_dir_all(&part).unwrap();
|
||||||
|
fs::write(
|
||||||
|
part.join(format!("doc{i:04}.txt")),
|
||||||
|
format!("commonneedle unique{i} rust document number {i}"),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
|
||||||
|
let start = Instant::now();
|
||||||
|
let added = index
|
||||||
|
.add_collection(&Collection {
|
||||||
|
name: "corpus".to_string(),
|
||||||
|
path: docs.display().to_string(),
|
||||||
|
shared: true,
|
||||||
|
exposure: EXPOSURE_FULL.to_string(),
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
println!("indexed {added} documents in {:?}", start.elapsed());
|
||||||
|
assert_eq!(added, total);
|
||||||
|
assert_eq!(index.doc_count(), total as u64);
|
||||||
|
|
||||||
|
let start = Instant::now();
|
||||||
|
let (hits, matches) = index.search("commonneedle", 10, true).unwrap();
|
||||||
|
println!("query matched {matches} documents in {:?}", start.elapsed());
|
||||||
|
assert_eq!(matches, total as u64);
|
||||||
|
assert_eq!(hits.len(), 10);
|
||||||
|
|
||||||
|
let response = build_response("scale", response_items(&hits), matches, 10);
|
||||||
|
assert!(response.truncated);
|
||||||
|
assert_eq!(response.more_available, (total - 10) as u64);
|
||||||
|
assert!(serde_json::to_string(&response).unwrap().len() > 0);
|
||||||
|
|
||||||
|
let (hits, matches) = index.search("unique777", 10, true).unwrap();
|
||||||
|
assert_eq!(matches, 1);
|
||||||
|
assert!(hits[0].summary.contains("unique777"));
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user