Add DESIGN.md: architecture, security posture, and decision log

AGENTS.md

@@ -2,6 +2,7 @@
 
 ## Repo shape
 - `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
+- `DESIGN.md` is the partner-facing architecture and decision log (with rationale for the spec cuts); keep it in sync when architecture decisions change.
 - `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
 - `DEPLOY.md` documents the TLS/deployment story: members need no TLS (outbound HTTPS), relays terminate TLS with Caddy or a tunnel, `ca_cert` adds private CAs, `allow_insecure` opts into plain http on private networks, and non-loopback `http://` is refused by default.
 - Commands: `cargo build`, `cargo test` (99 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.

DESIGN.md

@@ -0,0 +1,132 @@
+# FRX — Design and Decisions
+
+Status: Draft 0.5 (experimental), reference implementation `frxd` in Rust.
+This document is the architecture and decision record. The normative protocol surface is `rfc.txt`; deployment is `DEPLOY.md`; agent-facing notes are `AGENTS.md`.
+
+## 1. What FRX is
+
+FRX is a membership federation for retrieval. Members answer broadcast queries from content they already hold; there is no supply announcement stream and no in-protocol payment. The protocol standardizes the message layer and honesty constraints only: signed envelopes, budgets, honest truncation, egress consent, aggregate courtesy. Matching, relevance, ranking, retention, and trust are local.
+
+Any member may originate queries and answer them; roles are enable flags, never a deployment role (§I5).
+
+## 2. Components
+
+- **Member node (`frxd serve`)** — owns a keypair and an identifier, indexes local collections, broadcasts queries, answers queries from shared collections, receives responses. Local-first: local results are merged with remote results, provenance-marked.
+- **Relay (`frxd relay`)** — dumb, interchangeable transport. Holds no history, replays nothing, fans queries out to subscribed members, carries unicast responses/aggregates to member mailboxes. Relays may peer with each other to flood queries.
+- **Registry (MA)** — the membership authority: a signed, versioned snapshot listing identifiers, classes, authorized keys with validity windows, optional X25519 encryption keys, and relay endpoints. The registry is the sole authority for key-to-identifier binding.
+
+## 3. Message flow
+
+```
+querier --publish(signed query)--> relay A --flood--> relay B
+                                        |                 |
+                              mailbox fanout to all subscribed members
+                                        |                 |
+responder (on B) matches shared collections, signs response, encrypts to querier
+responder --unicast(signed ciphertext)--> relay network --> querier mailbox
+```
+
+- Queries are broadcast live to all members; silence is conformant and informative.
+- Responses are unicast to the querier and addressed by transport key.
+- Aggregates are bilateral, on request, per member, per period.
+- Members hold one authenticated SSE stream per relay (`/v1/stream`), with long-poll fallback on 404/405.
+- Relay federation is copy-only, hop-bounded, and duplicate-suppressed by envelope signature; direct publishes are never suppressed.
+
+## 4. Identity and trust
+
+- **Identifier**: an MA-hosted FQDN (`alice.frx.example`). No member-controlled DNS is required. Member-hosted identifiers (keys published in the member's own DNS, allowlisted by the MA) are planned, not normative.
+- **Credentials**: keys are rotatable and carry validity windows; multiple keys may be valid during rotation. Rotation publishes a successor before retiring the predecessor; revocation removes a key or shortens validity. A key never extends its own authority.
+- **Registry trust**: nodes pin the MA key. Snapshots are versioned (rollback rejected), signature-verified, and cached; registry outage fails static on the last validated snapshot. Open bootstrap requires an explicit development flag.
+- **Envelope authentication**: `{type, from, key, ts, nonce, body, sig}`; the signature covers the JCS (RFC 8785) canonical form of the unsigned envelope under a versioned prefix, and the receiver verifies both the signature and the registry binding `map[key].id == from`. Golden bytes and a deterministic signature are pinned in `tests/conformance.rs`.
+- **Freshness**: envelopes outside ±300 s are rejected. A node-side nonce cache is not implemented (replay inside the window is possible).
+
+## 5. Security posture
+
+What protects what:
+
+| Concern | Mechanism |
+| --- | --- |
+| Message authenticity | Ed25519 signature over the JCS envelope |
+| Key-to-identifier binding | MA-signed registry snapshot, pinned anchor |
+| Registry freshness | Monotonic version, fail-static cache |
+| Mailbox access | Challenge-response proof of key possession, single-use nonce |
+| Transport observation | TLS at relays (reverse proxy or tunnel), optional private CA |
+| Unicast confidentiality | X25519 / HKDF-SHA256 / ChaCha20-Poly1305 to the recipient's registry key |
+| Query confidentiality | None by design: receiver-local matching needs plaintext at members |
+
+Accepted limitations: relays see queries in clear by design (I3 constrains what may enter broadcasts); no end-to-end encryption is possible for queries; no directory-free admission on relays (optional gate); no bilateral node-side rate limiting yet; the local control API is unauthenticated and must stay on loopback.
+
+## 6. Economic and governance stance
+
+- Economics is out of protocol scope. The protocol carries no pricing, metering, settlement, citations, or receipts. Payments, licensing, and content transactions happen at the edge, on the owner's terms (e.g., `exposure: metadata` keeps content behind the owner's endpoint).
+- The MA governs identity, admission, and contract — who, never quality. Expulsion grounds are fabrication, admission fraud, and sustained abuse.
+- Off-wire conduct (link handling, retention, gating) is contractual; the protocol neither observes nor adjudicates it.
+- Aggregates advise only; they are inadmissible as sanction evidence. Defaults and relay governance (§10) remain the main soft-centralization risks.
+
+## 7. Decision log
+
+Decisions taken during design review, with rationale.
+
+| # | Decision | Rationale | Status |
+| --- | --- | --- | --- |
+| 1 | Minimal normative surface; all judgment local | Interop only needs the message layer; ranking/trust are local information problems | RFC §1–§2, §5 |
+| 2 | Role symmetry; querier/responder are flags | No privileged roles; one node may both ask and answer (I5) | Implemented |
+| 3 | Pull-only supply; no announce stream | A supply firehose adds cost and privacy exposure; queries already reach all members (I7, App. B) | RFC, implemented |
+| 4 | No citation/receipt economics | A retrieval protocol cannot observe citations on the web; self-issued artifacts have no trust anchor; removed in Draft 0.5 (App. B row) | Removed |
+| 5 | No scores on the wire; ordering/presentation local | Any ordering MUST is unfalsifiable without a standard scorer; scores invite open-ended comparability and reputation machinery | I6, §5 |
+| 6 | Eager/lazy retention removed from the spec | Content housekeeping is unobservable between peers; not an invariant | I5/§5 cleaned |
+| 7 | Honest truncation, not result-count etiquette | Quantity is querier-local, selection responder-local; budget plus a truncation flag suffice | §4, App. B |
+| 8 | Relay-mediated fanout with peer flooding; per-member isolation | O(1) publish; relays stay dumb and interchangeable; one lagging member must not stall the firehose | §3, implemented |
+| 9 | Visible backpressure, never silent drops | Lagging members get 429 + `missed` or an SSE `lag` event; publishers are unaffected | §3/§9, implemented |
+| 10 | Stable identifier + rotatable credentials; signed registry | Identity survives rotation; admission is gated once, credentials are self-managed; outage fails static | §4/§6, implemented |
+| 11 | `from` = identifier, `key` = pubkey; JCS canonical form | Resolves §10 canonicalization with a cross-language standard and a pinned golden vector | Implemented |
+| 12 | Relays address mailboxes by key, not by identifier | Keeps relays ignorant of identity and makes rotation local | Implemented |
+| 13 | SSE first, long-poll fallback | Push latency and connection efficiency; fallback for restricted networks | Implemented |
+| 14 | Unicast confidentiality profile (X25519/HKDF/ChaCha20-Poly1305) | Relays carry ciphertext; queries cannot be private (broadcast plus local matching) | Implemented, not yet normative |
+| 15 | Centralize coordination, localize judgment (I2) | Common state is cheaper held once: identity, admission, contract in the MA; matching, relevance, sharing, retention local | I2 reframed |
+| 16 | No sessions; per-message signatures | Peers are not connected; mailbox auth is a transport-local proof of possession | Implemented |
+| 17 | Lexical coverage gate before any rerank | Precision is project health; a demo false positive showed raw OR matching is too weak; embeddings later, local and replaceable | Implemented (`[match] min_coverage`) |
+
+## 8. Implementation status
+
+Built and tested (99 tests):
+
+- Envelope, JCS signing, registry binding, freshness window
+- Tantivy index, collections manifest, shared/exposure enforcement, reindex reset
+- Coverage-gated lexical matching
+- Query broadcast, SSE streaming, long-poll fallback, per-member queues and lag reporting
+- Relay federation, relay admission, registry watcher (path/URL, monotonic, fail-static)
+- Registry CLI (init/add/add-key/revoke-key/set-enc-key/set-relays/show/serve), key rotation
+- Aggregates (sent/passed, monthly floor, yearly rollup)
+- Encrypted unicast, TLS guardrails and custom CA support, static-build release profile
+
+Not built (see §9): dashboard UI, directory watching, user-supplied URL ingestion, node-side rate limiting, member-hosted identities, delegation, document lineage, MA anchor rollover, embedding rerank, invite-based self-enrollment, relay-to-relay unicast routing.
+
+## 9. Open issues
+
+From RFC §10 and implementation findings:
+
+- **Response routing across relays** — unicast is delivered on the relay where the recipient is subscribed; a responder whose configured relays do not include the recipient's relay cannot deliver. Current workaround: members connect to multiple relays. Relay-to-relay unicast forwarding is not implemented.
+- **Consumer admission tier** — automated/invite admission without weakening the Sybil defense.
+- **Default-relay governance** — registry-listed relays settle discovery; who operates the defaults remains a soft centralization point.
+- **Member-hosted identifiers** — keys in the member's own DNS instead of the MA registry.
+- **MA anchor rollover** — successor commitment and overlap for the registry signing key.
+- **Delegation** — granting authority to agents/sub-identities; unspecified.
+- **Document lineage** — revision/supersedes without a supply stream; unspecified.
+- **Unicast confidentiality profile** — implemented but not normative.
+- **Aggregate semantics** — counter definitions and the granularity floor are implemented choices from a terse spec; revisit with the sufficiency review.
+- **Replay** — ±300 s window only; no node nonce cache.
+- **Matching** — thresholds are untuned pending a real corpus; embedding rerank optional and local.
+
+## 10. Glossary
+
+- **Member** — an entity holding a keypair and a registry-listed identifier.
+- **Querier / responder** — the asking and answering roles of any member.
+- **Relay** — dumb transport that fans out broadcasts and holds member mailboxes.
+- **Registry / MA** — the membership authority and its signed snapshot.
+- **Envelope** — the signed message framing shared by all message types.
+- **Broadcast** — a query delivered to every subscribed member.
+- **Unicast** — a response or aggregate addressed to one member.
+- **Aggregate** — courtesy counters served bilaterally on request.
+- **Collection / shared / exposure** — local index unit; egress consent flag; metadata vs full content release.
+- **Coverage gate** — the minimum fraction of query terms a document must match to be a candidate.