56 lines
13 KiB
Markdown
56 lines
13 KiB
Markdown
# AGENTS.md
|
||
|
||
## Repo shape
|
||
- `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
|
||
- `DESIGN.md` is the partner-facing architecture and decision log (with rationale for the spec cuts); keep it in sync when architecture decisions change.
|
||
- `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
|
||
- `DEPLOY.md` documents the TLS/deployment story: members need no TLS (outbound HTTPS), relays terminate TLS with Caddy or a tunnel, `ca_cert` adds private CAs, `allow_insecure` opts into plain http on private networks, and non-loopback `http://` is refused by default.
|
||
- Commands: `cargo build`, `cargo test` (104 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config. unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
|
||
- E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.
|
||
|
||
## Editing the spec
|
||
- Read all of `rfc.txt` before editing; it is the sole source of truth and is deliberately terse.
|
||
- Keep the plain-text single-file format. Don't restructure into Markdown files unless asked.
|
||
- Invariants I1–I9 (§2) are normative; proposals contradicting them (scores in responses, topic taxonomy, announce stream, dispute messages, replayable broadcast, in-protocol pricing/settlement) are out of scope by design.
|
||
- Appendix B (Purge Log) is normative: a rejected mechanism may only be re-proposed if the written rationale is addressed.
|
||
- §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something.
|
||
- Use the spec's vocabulary — member/querier/responder, aggregates — not client/server or search-engine terms.
|
||
- A new `MUST` is only legitimate if it is observable at the boundary, deterministically verifiable by a peer, beneficial to the counterparty, and not derivable from local policy. Ranking/ordering/presentation fails this test and stays local (§5); scores never travel (I6).
|
||
- I2 is not blanket anti-centralization: shared coordination (identity, admission, contract) is centralized in the MA because common state is cheaper held once; decisions that consume local information (matching, relevance, sharing, retention) stay local. Off-wire conduct (link handling, retention, gating) is contract, not conformance.
|
||
|
||
## Implementation notes
|
||
- Wire format matches Draft 0.5: envelope `{type, from, key, ts, nonce, body, sig}` with `from` = identifier, `key` = pubkey; signatures cover the JCS canonical form of the unsigned envelope under prefix `FRX/0.5` (`src/crypto.rs`). Our `canonical_json` is JCS-compatible only for the restricted schema (ASCII keys, integers, no floats) — golden vectors in `tests/conformance.rs` pin the bytes and signature; revisit before claiming interop with non-Rust stacks.
|
||
- Relay addresses transport mailboxes by `key` (unicast `to` = recipient pubkey; queues keyed by pubkey); the identifier is protocol identity only. Registry binding checks `map[key].id == from`.
|
||
- Built: envelope/query/response, Tantivy index, aggregates, member directory. Not built: dashboard UI, directory watching, TLS, lineage/delegation. Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
|
||
- Economics is out of protocol scope (I4: aggregates advise, contracts govern): no receipt, citation, pricing, or settlement fields or message types exist or may be added.
|
||
- Relay verifies signatures and ±300s timestamp skew, carries only `query` broadcasts, holds no history (queues drained on poll), and requires challenge–response proof of key possession for mailbox polls. Nodes prefer SSE streams (`/v1/stream`, authenticated like polls) and fall back to long-poll on 404/405. Per-member queues are isolated: a lagging member gets 429 + Retry-After with a `missed` count; publishers and other members are never stalled. Relays MAY flood to configured peers (`/v1/federation`, `--peer`+`--url`, hop-bounded, seen-set dedup without suppressing identical direct publishes) and MAY gate senders against a registry (`--registry`+`--ma-key`).
|
||
- Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits results with honest `truncated`/`more_available` and no scores (I6). BM25 order is a local implementation detail, not protocol surface.
|
||
- Index layout: Tantivy at `<data_dir>/index`, collections manifest at `<data_dir>/collections.toml`; `exposure` (metadata|full) gates whether `content` is returned.
|
||
- Egress checks live in the responder path (`src/node.rs` `respond`), not the relay — keep private collections unreachable there.
|
||
- Appendix B is executable policy: `tests/purges.rs` has one absence test per rejected mechanism (12 rows). Add a negative test there before ever re-proposing one, and only if the rationale is addressed.
|
||
- `add`/`reindex` reset a collection (delete by manifest `name`) before re-adding, so deleted files don't linger; collection identity is its name, and same-named collections replace each other.
|
||
- Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale).
|
||
- Member authority (Draft 0.5 §6): the MA-signed registry snapshot is authoritative when configured (`[node] registry` = file path or URL, `ma_key` pinned; monotonic version — rollback and forgery close the node; file path is mtime-reloaded, URL is fetched at start + every 60s and cached to `<data_dir>/registry-cache.json`, so outage fails static). Keys carry optional validity windows (`not_before`/`not_after`); rotation = `registry add-key` then `revoke-key`.
|
||
- `<data_dir>/members.toml` (name, pubkey, `previous` keys, mtime-reloaded) is a dev/local fallback used only when no registry is configured; empty directory without a registry is open bootstrap only when `dev_bootstrap = true` (RFC §6: explicit dev flag).
|
||
- MA tooling: `frxd registry init|add|add-key|revoke-key|remove|list|applications|approve|invite|token|revoke-token|set-relays|show|serve` (signed `registry.json` + `ma-key.hex` in `--dir`); `frxd init --id/--registry/--ma-key`; `frxd key show|rotate`; `member add --previous <old>` for the fallback path. A node with no `[node] relays` discovers them from the registry snapshot (`doc.relays`).
|
||
- Aggregate semantics are our implementation choices from a terse spec: requests are `aggregate` envelopes carrying only `period`; replies carry `sent` (broadcasts that month) / `passed` (responses consumed from that member); granularity floor is enforced as YYYY or YYYY-MM only (finer rejected), yearly rolls up months. Revisit with §10 sufficiency review.
|
||
|
||
## Known gaps (Phase 2/3, intentional — don't fake them)
|
||
- No dashboard UI, no directory watching (new files need `reindex`), no TLS, no user-supplied URL ingestion, no node-side (bilateral) rate limiting.
|
||
- Unicast confidentiality is an implementation profile, not yet normative (RFC §10): response/aggregate bodies are encrypted to the recipient's registry-listed X25519 `enc_key` (`x25519-hkdf-sha256-chacha20poly1305`, `src/crypto.rs`); members without an `enc_key` get plaintext; relays verify signatures over ciphertext but cannot read payloads.
|
||
- Receipts/settlement are out of protocol scope, not unimplemented (I4; Appendix B row 2). Lineage and delegation remain §10 open issues — unspecified without a supply stream, so not buildable as written; don't invent them silently.
|
||
- Node query dedup is by `qid` only; replay inside the ±300s skew window remains possible (no nonce cache at nodes), relays have no directory/admission, and there is no end-to-end encryption — relays see everything in clear.
|
||
|
||
## Technical plans (deliberately not in the RFC)
|
||
- Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface.
|
||
- Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as ordinary members / backfill seeding; Appendix A names both as example derived corpora. Derived corpora are metadata-only via the member's own `exposure=metadata` collection setting (I9) — there is no registry-level class.
|
||
- Phase 1 (two-node query/response) is built and tested; the derived-corpora demo layers on top of it.
|
||
- Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate.
|
||
- frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher".
|
||
- Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role.
|
||
- Matching floor: boundary tokenizer (`src/tokenizer.rs` — letter/digit splits so `5555` matches `DLEX5555`, lowercase, ASCII fold, English stopwords+stemmer) → coverage gate (`[match] min_coverage`, default 0.4; 1–2 term queries require all terms) → title boost 2.0 + phrase boost 3.0 + query-time snippets. Schema changes require a fresh index dir (`open_or_create` errors on mismatch).
|
||
- Engine seam: `src/engine.rs` `SearchEngine` trait (`search` → `EngineOutput { hits, total: Option<u64> }`, `doc_count`); `respond()` in `src/node.rs` is the conformance wrapper (budget clamp, truncation from engine total — unknown total forces `truncated = true`). Power users can implement the trait (HTTP adapter or subprocess to an external engine).
|
||
- Onboarding: `frxd --onboarding` runs a wizard consuming a credential block (`id=.. token=.. registry=.. ma_key=..`) issued by the MA (`registry serve`; HTML page at `/`, `POST /v1/signup` queues a pending application, `POST /v1/enroll` binds keys and re-signs). Identity registration stays MA-side; the wizard never creates identities, only binds locally generated keys. Applications live in `<registry dir>/applications.json` (mode 600, MA contract data — never in the signed snapshot); `frxd registry approve <id>` promotes one (member stub + credential block whose token is the member's reusable account credential, hashed in `<registry dir>/tokens.json` — authorizes key enrollment for every node the member runs). `frxd registry token <id>` mints another member token, `revoke-token <id>` revokes all of a member's tokens; `frxd registry invite <id>` mints a single-use 24h handoff token (`<registry dir>/invites.json`). Prompts accept empty input as the default; scripted stdin works for tests.
|
||
- Next matching steps: eval harness with a small golden set (precision@k + false-silence rate), then a dense recall leg (model2vec-rs 0.2.1 exists but needs `default-features = false, features = ["fancy-regex", "local-only"]` for musl/airgapped; verify crate + model licenses before bundling), then an optional cross-encoder reranker. Embeddings are for recall; reranking is the precision tier.
|
||
- Identity/registry (RFC Draft 0.5 §4/§6): MA-hosted FQDN identifiers first (`<label>.frx.<ma-domain>`, no DNS needed by users), signed versioned registry snapshot with the MA key pinned; envelope `from` = identifier, `key` = pubkey; registry outage fails static. Member-hosted identities, MA anchor rollover, and unicast confidentiality are §10 open. Implementation phases: A (signed registry snapshot) and B (identifier + `key` + JCS on the wire) are built and tested. Prioritize frictionless onboarding (users may be department-level and cannot create DNS).
|