frxd/rfc.txt

FRX — Federated Retrieval Exchange

Status: Draft 0.5. Experimental. Reference implementation: frxd (Rust).

1. Summary

FRX is a membership federation for retrieval. Content owners answer broadcast queries. The protocol's normative surface is minimal: signed messages, budgets, truncation honesty, aggregate courtesy. Ranking, reputation, caching, filtering, verification, and quality judgment are node-local. There is no supply announcement stream: a responder answers only from content it already holds. A member's mandatory work per incoming query is a local lookup — nothing heavier is required.

2. Invariants

 I1 Ingress consent — content enters a member's index only via the publisher adding its own content or a user-supplied URL.
 I2 Local judgment — decisions that consume local information (matching, relevance, sharing, retention) are made locally. Shared coordination (identity, admission, contract) is centralized in the MA, because common state is cheaper held once.
 I3 Broadcast privacy — broadcast payloads MUST NOT contain third-party private content. Queries derived from scoring others' posts MUST be canonicalized (claims/entities). First-party user-initiated search text MAY be sent as typed.
 I4 Channel separation — statements advise (aggregates), contracts govern (membership); never cross-wired. The protocol carries no pricing, metering, or settlement.
 I5 Role symmetry — no privileged roles. Any member may originate queries or responses; no peer may require remote work per incoming query (I7). Neither role is privileged.
 I6 No scores — responses never carry numeric relevance; selection and presentation order are local policy (I2, §5).
 I7 Local-lookup cost — a member's mandatory work per incoming query is a local lookup. There is no supply stream to maintain, and no heavier mandatory work (fetch, embed, LLM). Cost scales with received query volume at lookup cost.
 I8 No shared vocabulary — no topic taxonomy; all filtering is receiver-local; the querier never classifies on behalf of receivers.
 I9 Egress consent — a member serves queries only from collections explicitly marked shared. Default is private.

3. System Model

Members exchange signed messages via dumb relays (multiply-operated transport, zero protocol authority). Queries are broadcast live to all members; relays hold no history and replay nothing. Relays MAY flood queries among themselves — copy-only, hop-bounded, duplicate-suppressed — so a member publishes once and coverage scales with relays, not senders. Delivery is per-member and isolated: a lagging member is backpressured visibly and never stalls publishers or peers. Responses are unicast from responder to querier. Transport DEFAULT: HTTPS + streaming (SSE) with long-poll fallback; members SHOULD hold one authenticated stream per relay. Bilateral rate limits with visible backpressure are normative; application-layer rejection is silent.

STREAM
TYPE
VISIBILITY
ORIGIN
Demand	query	Broadcast	Any member
Bid	response	Unicast to querier	Any member holding content
Courtesy	aggregate	Bilateral, on request	Querier
Silence is conformant and informative: an unanswered query means no member's available content produced a response — not that the content does not exist. A member holding matching content may still choose silence (I2, I5).

4. Messages

Envelope (all messages): {type, from, key, ts, nonce, body, sig}. `from` is the sender's member identifier; `key` is the Ed25519 public key used to sign; `sig` covers the canonical form of the other fields (RFC 8785 JCS under a versioned prefix; bodies carry no floating-point numbers). A receiver verifies the signature under `key`, then verifies that `key` is authorized for `from` by the member registry (§6).

query

jsonc

{ "qid": "...", "text": "<free-form, I3 applies>",
  "entities": ["Q192630"],            // optional, non-normative hint
  "budget": { "max_results": 5 } }
No format mandate. A query no receiver can match earns silence; intelligibility is enforced by retrieval economics, not conformance police (I8).

response

jsonc

{ "qid": "...",
  "results": [ { "url": "...", "title": "...", "summary": "...",
                 "published": "...", "exposure": "...", "content": null } ],
  "truncated": false, "more_available": 0, "cursor": null }
MUST NOT exceed max_results. MUST set truncated honestly if more matching results exist within budget (DNS TC-bit pattern). MUST NOT carry numeric relevance scores (I6). One result is a conformant, good response. exposure is "metadata" | "full" (paywall compatibility); content is present only when exposure is "full". Republication of another member's response is a contract matter.

aggregate (on request, per member, per period)

jsonc

 { "period": "2026-03", "sent": 12400, "passed": 310 }
Counters are the querier's own; no dispute or appeal messages exist, and none may be added (I4). Granularity floor is normative.

There is no publish/announce message. Document metadata is carried in responses (above). Document lineage (revision/supersedes) and delegation to an indexer are unspecified without a supply stream; see §10.

5. Local Policy Domains

Protocol-silent by design (I2): ranking, ordering, presentation, relevance gating, reputation counters and throttles, verification/spot-checks, caching and invalidation, claim minting, external fallback, sharing policy above the I9 floor. Advisory reputation bureaus MAY exist; no member is bound.

6. Membership

The MA governs identity, contract, expulsion — who, never quality. Identifiers are MA-hosted FQDNs (`<label>.frx.<ma-domain>`); no member-controlled DNS is required. Member-hosted identifiers — keys published in the member's own domain and allowlisted by the MA — are planned, not yet normative. The MA maintains a signed, versioned registry snapshot listing identifiers, class, authorized keys with validity windows, an optional X25519 encryption key per member, and the federation's relay endpoints. Members and queriers may discover relays from it; relays MAY verify sender admission against it, rejecting unlisted keys visibly. Nodes pin the MA key; the snapshot is the sole authority for the key→identifier binding. Rotation publishes a successor key before retiring its predecessor; revocation removes a key or shortens its validity. Registry outage is fail-static: the last validated snapshot stays in force, and open bootstrap requires an explicit development flag. Admission cost is the Sybil defense. Expulsion grounds: fabrication, admission fraud, sustained abuse — never low quality. Escalation: local throttle → advisory aggregates → MA warning → delisting → expulsion. Aggregates are inadmissible as sanction evidence (I4). Conduct not observable on the wire — link handling, retention, gating — is governed by contract; the protocol neither observes nor adjudicates it. Membership classes: source members (own content) and enrichment members (derived corpora, e.g. GDELT/CC-NEWS bots — metadata-only exposure, transformation logic open and auditable).

7. Reference Implementation — frxd

A single static Rust binary. Install, point at a directory, done. It is simultaneously: (a) a personal search engine over local files, (b) a conformant FRX member.

First run: generate keypair, join the MA registry (no domain or DNS required), write config.toml, open localhost web UI (plus frx search CLI). Index target directories with Tantivy; watch for changes; extract text from txt/md/html (PDF optional). Nothing is shared until a collection is explicitly marked shared (I9).

As responder: subscribe to the live query stream via configured relays; match incoming queries against shared collections only (receiver-local lexical/embedding match); respond within budget with honest truncation; sign.

As consumer: search is local-first; a network toggle broadcasts the query and merges responses, provenance-marked ("your files" / "member X").

Dashboard: sent / passed per period (self-derived from local counters; aggregates requested from queriers on demand).

Stack: tokio, tantivy, axum (localhost UI), reqwest, ed25519-dalek, notify. One binary; frxd relay runs a relay for sovereignty-minded users.

Build order: Phase 1 — envelope, query stream, query, response (demoable between two nodes; consumer packaging is a skin over this). Phase 2 — aggregates, dashboard. Phase 3 — lineage, delegation (app layer).

8. Security & Privacy Considerations

Query visibility is total among members; abstraction level and membership are the boundary (I3). Derived queries SHOULD minimize personal data (I3); each member is responsible for the content of its own messages. Response streams are strategic disclosure (corpus mapping, intake intelligence) — unicast, need-to-know. Amplification is bounded by bilateral transport limits and contract, not routing. Publisher self-promotion is the expected adversarial mode; defense is local (gate, pass-rate throttle, local source reputation). Enrichment members' filters are an editorial power — auditable openness is the mitigation.

9. Conformance

A conforming implementation: signs all messages with a key authorized for its identifier in the member registry; publishes queries to the firehose only; respects I3 and I9; respects max_results; truncates honestly; sends results without scores; enforces visible transport backpressure, never silent transport drops; serves aggregates on request at or above the granularity floor; ingests content only via publisher-added content or user-supplied URL; implements no dispute messages.

10. Open Issues

Consumer admission tier — automated/invite admission for distributed binaries without weakening the Sybil defense (MA policy, gates §7 adoption).
Default-relay governance — registry-listed relays settle discovery, but who operates the defaults remains a soft centralization point (mitigate with multiple operators + one-command self-host).
Delegation grant mechanism.
Member-hosted identifiers — keys published in the member's own DNS instead of the MA registry; MA-hosted is normative until specified.
MA anchor rollover — successor commitment and overlap for the registry signing key.
Unicast confidentiality — the reference implementation encrypts response and aggregate bodies to the recipient's registry-listed X25519 key (X25519 / HKDF-SHA256 / ChaCha20-Poly1305); the profile is not yet normative.
Document lineage (revision/supersedes) and delegation without a supply stream — previously carried by publish; now unspecified.
Claim/entity minting conventions — recommended, non-normative (I8).

Appendix A. Recommended Local Practices (Non-Normative)

Two-stage ingestion (cheap gate before any LLM attention); per-peer sent/passed counters, throttling on ratios; verdict memoization and claim normalization; hot-set replication; backfill seeding (Wikipedia/Wikidata, GDELT, CC-NEWS); measure query-to-claim collapse and domain concentration before sizing anything.

Appendix B. Purge Log (Normative)

Re-proposals MUST address the rationale.

MECHANISM
REJECTED BECAUSE
Per-query bounty, winner-selection, slashing	Pays for query execution when responders own content; race-to-first rewards speed over honesty; anonymous-trust machinery with no anonymous peers
In-protocol citation accounting / receipts as settlement	Economics is not protocol surface; a retrieval protocol cannot observe citations on the web, and self-issued artifacts have no trust anchor
Protocol query dedup	Undecidable at network layer; duplicates cost receivers ~nothing; dedup is local caching
k-fetch ingestion attestations	Swarm trust machinery; signatures + local spot-checks suffice
Result-count etiquette (SERP min/max)	Rendering convention from the one-actor world; quantity is querier-local, selection responder-local; budget + truncation flag suffice
Global reputation score	Relevance is locally defined; consumption is observable where it happens
Aggregate appeals	Converts courtesy into litigation; drags local policy into network process
Topic channels	Shared vocabulary is a governance object; sender-side routing by the least-informed party yields undetectable false negatives; at membership N, broadcast + receiver filtering is cheaper than the coordination
Broadcast responses ("evidence commons")	Multiplies the heavy stream by N; requires network-layer query equivalence (purged); unicast need-to-know suffices
Normative query canonical form	Receiver-local matching handles arbitrary phrasing; intelligibility is self-enforcing; the protocol constrains only what must not enter a broadcast (I3)
Supply announce firehose	Pull-only supply: responders answer from content already held; a visible announce stream adds cost and privacy exposure with no discovery benefit, since queries already reach all members
Durable / replayable broadcast stream	A live, ephemeral broadcast suffices for routing; persistence and history-replay buy nothing and invite retention and erasure problems
Two closing notes. First, the open issue that matters most is consumer admission — everything else in §7 is straightforward engineering, but "naive user installs a binary and joins a membership org" has real tension with the admission-cost Sybil defense, and I'd resolve it deliberately rather than by accident (invite codes at launch; a lightweight consumer tier later; or defaulting consumer nodes to delegation through an indexer member until they've earned standing). Second, when the reference implementation exists, the purge log stops being documentation and becomes a test suite: one unit test per row, each asserting the absence of a mechanism. A spec this small can afford to test what it refuses to do.