SSE streaming with long-poll fallback; encrypted unicast profile; registry enc keys

2026-09-15 06:58:35 -04:00
parent 2c97fd523f
commit ec98275613
15 changed files with 1117 additions and 320 deletions
@@ -3,7 +3,7 @@
 ## Repo shape
 - `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
 - `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
- Commands: `cargo build`, `cargo test` (85 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
+- Commands: `cargo build`, `cargo test` (90 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
 - E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.

 ## Editing the spec
@@ -21,7 +21,7 @@
 - Relay addresses transport mailboxes by `key` (unicast `to` = recipient pubkey; queues keyed by pubkey); the identifier is protocol identity only. Registry binding checks `map[key].id == from`.
 - Built: envelope/query/response, Tantivy index, aggregates, member directory. Not built: dashboard UI, directory watching, TLS, lineage/delegation. Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
 - Economics is out of protocol scope (I4: aggregates advise, contracts govern): no receipt, citation, pricing, or settlement fields or message types exist or may be added.
- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queues drained on poll), and requires challenge–response proof of key possession for mailbox polls. Per-member queues are isolated: a lagging member gets 429 + Retry-After with a `missed` count; publishers and other members are never stalled. Relays MAY flood to configured peers (`/v1/federation`, `--peer`+`--url`, hop-bounded, seen-set dedup without suppressing identical direct publishes) and MAY gate senders against a registry (`--registry`+`--ma-key`).
+- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queues drained on poll), and requires challenge–response proof of key possession for mailbox polls. Nodes prefer SSE streams (`/v1/stream`, authenticated like polls) and fall back to long-poll on 404/405. Per-member queues are isolated: a lagging member gets 429 + Retry-After with a `missed` count; publishers and other members are never stalled. Relays MAY flood to configured peers (`/v1/federation`, `--peer`+`--url`, hop-bounded, seen-set dedup without suppressing identical direct publishes) and MAY gate senders against a registry (`--registry`+`--ma-key`).
 - Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits results with honest `truncated`/`more_available` and no scores (I6). BM25 order is a local implementation detail, not protocol surface.
 - Index layout: Tantivy at `<data_dir>/index`, collections manifest at `<data_dir>/collections.toml`; `exposure` (metadata|full) gates whether `content` is returned.
 - Egress checks live in the responder path (`src/node.rs` `respond`), not the relay — keep private collections unreachable there.
@@ -34,7 +34,8 @@
 - Aggregate semantics are our implementation choices from a terse spec: requests are `aggregate` envelopes carrying only `period`; replies carry `sent` (broadcasts that month) / `passed` (responses consumed from that member); granularity floor is enforced as YYYY or YYYY-MM only (finer rejected), yearly rolls up months. Revisit with §10 sufficiency review.

 ## Known gaps (Phase 2/3, intentional — don't fake them)
- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting, no SSE streaming (long-poll only), no unicast confidentiality (HPKE).
+- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting.
+- Unicast confidentiality is an implementation profile, not yet normative (RFC §10): response/aggregate bodies are encrypted to the recipient's registry-listed X25519 `enc_key` (`x25519-hkdf-sha256-chacha20poly1305`, `src/crypto.rs`); members without an `enc_key` get plaintext; relays verify signatures over ciphertext but cannot read payloads.
 - Receipts/settlement are out of protocol scope, not unimplemented (I4; Appendix B row 2). Lineage and delegation remain §10 open issues — unspecified without a supply stream, so not buildable as written; don't invent them silently.
 - Node query dedup is by `qid` only; replay inside the ±300s skew window remains possible (no nonce cache at nodes), relays have no directory/admission, and there is no end-to-end encryption — relays see everything in clear.