frxd/DESIGN.md

# FRX — Design and Decisions

Status: Draft 0.5 (experimental), reference implementation `frxd` in Rust.
This document is the architecture and decision record. The normative protocol surface is `rfc.txt`; deployment is `DEPLOY.md`; agent-facing notes are `AGENTS.md`.

## 1. What FRX is

FRX is a membership federation for retrieval. Members answer broadcast queries from content they already hold; there is no supply announcement stream and no in-protocol payment. The protocol standardizes the message layer and honesty constraints only: signed envelopes, budgets, honest truncation, egress consent, aggregate courtesy. Matching, relevance, ranking, retention, and trust are local.

Any member may originate queries and answer them; roles are enable flags, never a deployment role (§I5).

## 2. Components

- **Member node (`frxd serve`)** — owns a keypair and an identifier, indexes local collections, broadcasts queries, answers queries from shared collections, receives responses. Local-first: local results are merged with remote results, provenance-marked.
- **Relay (`frxd relay`)** — dumb, interchangeable transport. Holds no history, replays nothing, fans queries out to subscribed members, carries unicast responses/aggregates to member mailboxes. Relays may peer with each other to flood queries.
- **Registry (MA)** — the membership authority: a signed, versioned snapshot listing identifiers, authorized keys with validity windows, optional X25519 encryption keys, and relay endpoints. The registry is the sole authority for key-to-identifier binding.

## 3. Message flow

```
querier --publish(signed query)--> relay A --flood--> relay B
                                        |                 |
                              mailbox fanout to all subscribed members
                                        |                 |
responder (on B) matches shared collections, signs response, encrypts to querier
responder --unicast(signed ciphertext)--> relay network --> querier mailbox
```

- Queries are broadcast live to all members; silence is conformant and informative.
- Responses are unicast to the querier and addressed by transport key.
- Aggregates are bilateral, on request, per member, per period.
- Members hold one authenticated SSE stream per relay (`/v1/stream`), with long-poll fallback on 404/405.
- Relay federation is copy-only, hop-bounded, and duplicate-suppressed by envelope signature; direct publishes are never suppressed.

## 4. Identity and trust

- **Identifier**: an MA-hosted FQDN (`alice.frx.federatedsearch.org`). No member-controlled DNS is required. Member-hosted identifiers (keys published in the member's own DNS, allowlisted by the MA) are planned, not normative.
- **Credentials**: keys are rotatable and carry validity windows; multiple keys may be valid during rotation. Rotation publishes a successor before retiring the predecessor; revocation removes a key or shortens validity. A key never extends its own authority.
- **Registry trust**: nodes pin the MA key. Snapshots are versioned (rollback rejected), signature-verified, and cached; registry outage fails static on the last validated snapshot. Open bootstrap requires an explicit development flag.
- **Envelope authentication**: `{type, from, key, ts, nonce, body, sig}`; the signature covers the JCS (RFC 8785) canonical form of the unsigned envelope under a versioned prefix, and the receiver verifies both the signature and the registry binding `map[key].id == from`. Golden bytes and a deterministic signature are pinned in `tests/conformance.rs`.
- **Freshness**: envelopes outside ±300 s are rejected. A node-side nonce cache is not implemented (replay inside the window is possible).

## 5. Security posture

What protects what:

| Concern | Mechanism |
| --- | --- |
| Message authenticity | Ed25519 signature over the JCS envelope |
| Key-to-identifier binding | MA-signed registry snapshot, pinned anchor |
| Registry freshness | Monotonic version, fail-static cache |
| Mailbox access | Challenge-response proof of key possession, single-use nonce |
| Transport observation | TLS at relays (reverse proxy or tunnel), optional private CA |
| Unicast confidentiality | X25519 / HKDF-SHA256 / ChaCha20-Poly1305 to the recipient's registry key |
| Query confidentiality | None by design: receiver-local matching needs plaintext at members |

Accepted limitations: relays see queries in clear by design (I3 constrains what may enter broadcasts); no end-to-end encryption is possible for queries; no directory-free admission on relays (optional gate); no bilateral node-side rate limiting yet; the local control API is unauthenticated and must stay on loopback.

## 6. Economic and governance stance

- Economics is out of protocol scope. The protocol carries no pricing, metering, settlement, citations, or receipts. Payments, licensing, and content transactions happen at the edge, on the owner's terms (e.g., `exposure: metadata` keeps content behind the owner's endpoint).
- The MA governs identity, admission, and contract — who, never quality. Expulsion grounds are fabrication, admission fraud, and sustained abuse.
- Off-wire conduct (link handling, retention, gating) is contractual; the protocol neither observes nor adjudicates it.
- Aggregates advise only; they are inadmissible as sanction evidence. Defaults and relay governance (§10) remain the main soft-centralization risks.

## 7. Decision log

Decisions taken during design review, with rationale.

| # | Decision | Rationale | Status |
| --- | --- | --- | --- |
| 1 | Minimal normative surface; all judgment local | Interop only needs the message layer; ranking/trust are local information problems | RFC §1–§2, §5 |
| 2 | Role symmetry; querier/responder are flags | No privileged roles; one node may both ask and answer (I5) | Implemented |
| 3 | Pull-only supply; no announce stream | A supply firehose adds cost and privacy exposure; queries already reach all members (I7, App. B) | RFC, implemented |
| 4 | No citation/receipt economics | A retrieval protocol cannot observe citations on the web; self-issued artifacts have no trust anchor; removed in Draft 0.5 (App. B row) | Removed |
| 5 | No scores on the wire; ordering/presentation local | Any ordering MUST is unfalsifiable without a standard scorer; scores invite open-ended comparability and reputation machinery | I6, §5 |
| 6 | Eager/lazy retention removed from the spec | Content housekeeping is unobservable between peers; not an invariant | I5/§5 cleaned |
| 7 | Honest truncation, not result-count etiquette | Quantity is querier-local, selection responder-local; budget plus a truncation flag suffice | §4, App. B |
| 8 | Relay-mediated fanout with peer flooding; per-member isolation | O(1) publish; relays stay dumb and interchangeable; one lagging member must not stall the firehose | §3, implemented |
| 9 | Visible backpressure, never silent drops | Lagging members get 429 + `missed` or an SSE `lag` event; publishers are unaffected | §3/§9, implemented |
| 10 | Stable identifier + rotatable credentials; signed registry | Identity survives rotation; admission is gated once, credentials are self-managed; outage fails static | §4/§6, implemented |
| 11 | `from` = identifier, `key` = pubkey; JCS canonical form | Resolves §10 canonicalization with a cross-language standard and a pinned golden vector | Implemented |
| 12 | Relays address mailboxes by key, not by identifier | Keeps relays ignorant of identity and makes rotation local | Implemented |
| 13 | SSE first, long-poll fallback | Push latency and connection efficiency; fallback for restricted networks | Implemented |
| 14 | Unicast confidentiality profile (X25519/HKDF/ChaCha20-Poly1305) | Relays carry ciphertext; queries cannot be private (broadcast plus local matching) | Implemented, not yet normative |
| 15 | Centralize coordination, localize judgment (I2) | Common state is cheaper held once: identity, admission, contract in the MA; matching, relevance, sharing, retention local | I2 reframed |
| 16 | No sessions; per-message signatures | Peers are not connected; mailbox auth is a transport-local proof of possession | Implemented |
| 17 | Lexical coverage gate before any rerank | Precision is project health; a demo false positive showed raw OR matching is too weak; embeddings later, local and replaceable | Implemented (`[match] min_coverage`) |
| 18 | Default engine: boundary tokenizer + fold + stopwords + stemmer, title/phrase boosts, query-time snippets | The floor must be high out of the box; model-number and morphology matching are cheap wins with no model | Implemented (`src/tokenizer.rs`) |
| 19 | Engine seam: `SearchEngine` trait with the responder path as the conformance wrapper | Plugins can change quality, never conformance; engines return `Option<total>` so an external engine can't fake the truncation bit | Implemented (`src/engine.rs`) |
| 20 | Onboarding is a wizard consuming MA-issued credentials; signup lives on the MA's site | Users may be department-level and cannot create domains or DNS records; the wizard never creates identities, only binds locally generated keys | Implemented (`frxd --onboarding`, `/v1/signup`+`/v1/enroll`) |

## 8. Implementation status

Built and tested (99 tests):

- Envelope, JCS signing, registry binding, freshness window
- Tantivy index, collections manifest, shared/exposure enforcement, reindex reset
- Coverage-gated lexical matching with boundary tokenization, stemming, folding, boosts, and snippets
- Engine seam (`SearchEngine` trait) behind the conformance wrapper
- Query broadcast, SSE streaming, long-poll fallback, per-member queues and lag reporting
- Relay federation, relay admission, registry watcher (path/URL, monotonic, fail-static)
- Registry CLI (init/add/add-key/revoke-key/set-enc-key/set-relays/show/serve), key rotation
- Aggregates (sent/passed, monthly floor, yearly rollup)
- Encrypted unicast, TLS guardrails and custom CA support, static-build release profile
- Onboarding wizard (`frxd --onboarding`) and MA signup/enroll site (`/v1/signup`, `/v1/enroll`, HTML at `/`)

Not built (see §9): dashboard UI, directory watching, user-supplied URL ingestion, node-side rate limiting, member-hosted identities, delegation, document lineage, MA anchor rollover, embedding rerank, invite-based self-enrollment, relay-to-relay unicast routing.

## 9. Open issues

From RFC §10 and implementation findings:

- **Response routing across relays** — unicast is delivered on the relay where the recipient is subscribed; a responder whose configured relays do not include the recipient's relay cannot deliver. Current workaround: members connect to multiple relays. Relay-to-relay unicast forwarding is not implemented.
- **Consumer admission tier** — automated/invite admission without weakening the Sybil defense.
- **Default-relay governance** — registry-listed relays settle discovery; who operates the defaults remains a soft centralization point.
- **Member-hosted identifiers** — keys in the member's own DNS instead of the MA registry.
- **MA anchor rollover** — successor commitment and overlap for the registry signing key.
- **Delegation** — granting authority to agents/sub-identities; unspecified.
- **Document lineage** — revision/supersedes without a supply stream; unspecified.
- **Unicast confidentiality profile** — implemented but not normative.
- **Aggregate semantics** — counter definitions and the granularity floor are implemented choices from a terse spec; revisit with the sufficiency review.
- **Replay** — ±300 s window only; no node nonce cache.
- **Matching** — thresholds are untuned pending a real corpus; embedding rerank optional and local.

## 10. Glossary

- **Member** — an entity holding a keypair and a registry-listed identifier.
- **Querier / responder** — the asking and answering roles of any member.
- **Relay** — dumb transport that fans out broadcasts and holds member mailboxes.
- **Registry / MA** — the membership authority and its signed snapshot.
- **Envelope** — the signed message framing shared by all message types.
- **Broadcast** — a query delivered to every subscribed member.
- **Unicast** — a response or aggregate addressed to one member.
- **Aggregate** — courtesy counters served bilaterally on request.
- **Collection / shared / exposure** — local index unit; egress consent flag; metadata vs full content release.
- **Coverage gate** — the minimum fraction of query terms a document must match to be a candidate.