frxd/DESIGN.md

FRX — Design and Decisions

Status: Draft 0.5 (experimental), reference implementation frxd in Rust. This document is the architecture and decision record. The normative protocol surface is rfc.txt; deployment is DEPLOY.md; agent-facing notes are AGENTS.md.

1. What FRX is

FRX is a membership federation for retrieval. Members answer broadcast queries from content they already hold; there is no supply announcement stream and no in-protocol payment. The protocol standardizes the message layer and honesty constraints only: signed envelopes, budgets, honest truncation, egress consent, aggregate courtesy. Matching, relevance, ranking, retention, and trust are local.

Any member may originate queries and answer them; roles are enable flags, never a deployment role (§I5).

2. Components

• Member node (frxd serve) — owns a keypair and an identifier, indexes local collections, broadcasts queries, answers queries from shared collections, receives responses. Local-first: local results are merged with remote results, provenance-marked.
• Relay (frxd relay) — dumb, interchangeable transport. Holds no history, replays nothing, fans queries out to subscribed members, carries unicast responses/aggregates to member mailboxes. Relays may peer with each other to flood queries.
• Registry (MA) — the membership authority: a signed, versioned snapshot listing identifiers, classes, authorized keys with validity windows, optional X25519 encryption keys, and relay endpoints. The registry is the sole authority for key-to-identifier binding.

3. Message flow

querier --publish(signed query)--> relay A --flood--> relay B
                                        |                 |
                              mailbox fanout to all subscribed members
                                        |                 |
responder (on B) matches shared collections, signs response, encrypts to querier
responder --unicast(signed ciphertext)--> relay network --> querier mailbox

• Queries are broadcast live to all members; silence is conformant and informative.
• Responses are unicast to the querier and addressed by transport key.
• Aggregates are bilateral, on request, per member, per period.
• Members hold one authenticated SSE stream per relay (/v1/stream), with long-poll fallback on 404/405.
• Relay federation is copy-only, hop-bounded, and duplicate-suppressed by envelope signature; direct publishes are never suppressed.

4. Identity and trust

• Identifier: an MA-hosted FQDN (alice.frx.federatedsearch.org). No member-controlled DNS is required. Member-hosted identifiers (keys published in the member's own DNS, allowlisted by the MA) are planned, not normative.
• Credentials: keys are rotatable and carry validity windows; multiple keys may be valid during rotation. Rotation publishes a successor before retiring the predecessor; revocation removes a key or shortens validity. A key never extends its own authority.
• Registry trust: nodes pin the MA key. Snapshots are versioned (rollback rejected), signature-verified, and cached; registry outage fails static on the last validated snapshot. Open bootstrap requires an explicit development flag.
• Envelope authentication: {type, from, key, ts, nonce, body, sig}; the signature covers the JCS (RFC 8785) canonical form of the unsigned envelope under a versioned prefix, and the receiver verifies both the signature and the registry binding map[key].id == from. Golden bytes and a deterministic signature are pinned in tests/conformance.rs.
• Freshness: envelopes outside ±300 s are rejected. A node-side nonce cache is not implemented (replay inside the window is possible).

5. Security posture

What protects what:

Concern	Mechanism
Message authenticity	Ed25519 signature over the JCS envelope
Key-to-identifier binding	MA-signed registry snapshot, pinned anchor
Registry freshness	Monotonic version, fail-static cache
Mailbox access	Challenge-response proof of key possession, single-use nonce
Transport observation	TLS at relays (reverse proxy or tunnel), optional private CA
Unicast confidentiality	X25519 / HKDF-SHA256 / ChaCha20-Poly1305 to the recipient's registry key
Query confidentiality	None by design: receiver-local matching needs plaintext at members

Accepted limitations: relays see queries in clear by design (I3 constrains what may enter broadcasts); no end-to-end encryption is possible for queries; no directory-free admission on relays (optional gate); no bilateral node-side rate limiting yet; the local control API is unauthenticated and must stay on loopback.

6. Economic and governance stance

• Economics is out of protocol scope. The protocol carries no pricing, metering, settlement, citations, or receipts. Payments, licensing, and content transactions happen at the edge, on the owner's terms (e.g., exposure: metadata keeps content behind the owner's endpoint).
• The MA governs identity, admission, and contract — who, never quality. Expulsion grounds are fabrication, admission fraud, and sustained abuse.
• Off-wire conduct (link handling, retention, gating) is contractual; the protocol neither observes nor adjudicates it.
• Aggregates advise only; they are inadmissible as sanction evidence. Defaults and relay governance (§10) remain the main soft-centralization risks.

7. Decision log

Decisions taken during design review, with rationale.

#	Decision	Rationale	Status
1	Minimal normative surface; all judgment local	Interop only needs the message layer; ranking/trust are local information problems	RFC §1–§2, §5
2	Role symmetry; querier/responder are flags	No privileged roles; one node may both ask and answer (I5)	Implemented
3	Pull-only supply; no announce stream	A supply firehose adds cost and privacy exposure; queries already reach all members (I7, App. B)	RFC, implemented
4	No citation/receipt economics	A retrieval protocol cannot observe citations on the web; self-issued artifacts have no trust anchor; removed in Draft 0.5 (App. B row)	Removed
5	No scores on the wire; ordering/presentation local	Any ordering MUST is unfalsifiable without a standard scorer; scores invite open-ended comparability and reputation machinery	I6, §5
6	Eager/lazy retention removed from the spec	Content housekeeping is unobservable between peers; not an invariant	I5/§5 cleaned
7	Honest truncation, not result-count etiquette	Quantity is querier-local, selection responder-local; budget plus a truncation flag suffice	§4, App. B
8	Relay-mediated fanout with peer flooding; per-member isolation	O(1) publish; relays stay dumb and interchangeable; one lagging member must not stall the firehose	§3, implemented
9	Visible backpressure, never silent drops	Lagging members get 429 + missed or an SSE lag event; publishers are unaffected	§3/§9, implemented
10	Stable identifier + rotatable credentials; signed registry	Identity survives rotation; admission is gated once, credentials are self-managed; outage fails static	§4/§6, implemented
11	from = identifier, key = pubkey; JCS canonical form	Resolves §10 canonicalization with a cross-language standard and a pinned golden vector	Implemented
12	Relays address mailboxes by key, not by identifier	Keeps relays ignorant of identity and makes rotation local	Implemented
13	SSE first, long-poll fallback	Push latency and connection efficiency; fallback for restricted networks	Implemented
14	Unicast confidentiality profile (X25519/HKDF/ChaCha20-Poly1305)	Relays carry ciphertext; queries cannot be private (broadcast plus local matching)	Implemented, not yet normative
15	Centralize coordination, localize judgment (I2)	Common state is cheaper held once: identity, admission, contract in the MA; matching, relevance, sharing, retention local	I2 reframed
16	No sessions; per-message signatures	Peers are not connected; mailbox auth is a transport-local proof of possession	Implemented
17	Lexical coverage gate before any rerank	Precision is project health; a demo false positive showed raw OR matching is too weak; embeddings later, local and replaceable	Implemented ([match] min_coverage)
18	Default engine: boundary tokenizer + fold + stopwords + stemmer, title/phrase boosts, query-time snippets	The floor must be high out of the box; model-number and morphology matching are cheap wins with no model	Implemented (src/tokenizer.rs)
19	Engine seam: SearchEngine trait with the responder path as the conformance wrapper	Plugins can change quality, never conformance; engines return Option<total> so an external engine can't fake the truncation bit	Implemented (src/engine.rs)
20	Onboarding is a wizard consuming MA-issued credentials; signup lives on the MA's site	Users may be department-level and cannot create domains or DNS records; the wizard never creates identities, only binds locally generated keys	Implemented (frxd --onboarding, /v1/signup+/v1/enroll)

8. Implementation status

Built and tested (99 tests):

• Envelope, JCS signing, registry binding, freshness window
• Tantivy index, collections manifest, shared/exposure enforcement, reindex reset
• Coverage-gated lexical matching with boundary tokenization, stemming, folding, boosts, and snippets
• Engine seam (SearchEngine trait) behind the conformance wrapper
• Query broadcast, SSE streaming, long-poll fallback, per-member queues and lag reporting
• Relay federation, relay admission, registry watcher (path/URL, monotonic, fail-static)
• Registry CLI (init/add/add-key/revoke-key/set-enc-key/set-relays/show/serve), key rotation
• Aggregates (sent/passed, monthly floor, yearly rollup)
• Encrypted unicast, TLS guardrails and custom CA support, static-build release profile
• Onboarding wizard (frxd --onboarding) and MA signup/enroll site (/v1/signup, /v1/enroll, HTML at /)

Not built (see §9): dashboard UI, directory watching, user-supplied URL ingestion, node-side rate limiting, member-hosted identities, delegation, document lineage, MA anchor rollover, embedding rerank, invite-based self-enrollment, relay-to-relay unicast routing.

9. Open issues

From RFC §10 and implementation findings:

• Response routing across relays — unicast is delivered on the relay where the recipient is subscribed; a responder whose configured relays do not include the recipient's relay cannot deliver. Current workaround: members connect to multiple relays. Relay-to-relay unicast forwarding is not implemented.
• Consumer admission tier — automated/invite admission without weakening the Sybil defense.
• Default-relay governance — registry-listed relays settle discovery; who operates the defaults remains a soft centralization point.
• Member-hosted identifiers — keys in the member's own DNS instead of the MA registry.
• MA anchor rollover — successor commitment and overlap for the registry signing key.
• Delegation — granting authority to agents/sub-identities; unspecified.
• Document lineage — revision/supersedes without a supply stream; unspecified.
• Unicast confidentiality profile — implemented but not normative.
• Aggregate semantics — counter definitions and the granularity floor are implemented choices from a terse spec; revisit with the sufficiency review.
• Replay — ±300 s window only; no node nonce cache.
• Matching — thresholds are untuned pending a real corpus; embedding rerank optional and local.

10. Glossary

• Member — an entity holding a keypair and a registry-listed identifier.
• Querier / responder — the asking and answering roles of any member.
• Relay — dumb transport that fans out broadcasts and holds member mailboxes.
• Registry / MA — the membership authority and its signed snapshot.
• Envelope — the signed message framing shared by all message types.
• Broadcast — a query delivered to every subscribed member.
• Unicast — a response or aggregate addressed to one member.
• Aggregate — courtesy counters served bilaterally on request.
• Collection / shared / exposure — local index unit; egress consent flag; metadata vs full content release.
• Coverage gate — the minimum fraction of query terms a document must match to be a candidate.