frxd/AGENTS.md

AGENTS.md

Repo shape

• rfc.txt (FRX — Federated Retrieval Exchange, Draft 0.3) is the normative spec; src/ is the Phase 1 frxd implementation (single crate, two binaries).
• frxd is the member node (init/add/index/serve/relay/query/status); frx is the thin client (search/query/status). Relay and node roles are separate subcommands.
• Commands: cargo build, cargo test (62 tests: unit in src/; e2e tests/phase1.rs; conformance tests/conformance.rs; concurrency/restart tests/concurrency.rs; real subprocess CLI tests/cli.rs; 1000-doc tests/scale.rs; purge-log absence tests/purges.rs; shared fixtures tests/common/mod.rs). No CI/lint config.
• E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use tests/common/mod.rs helpers (spawn_relay*, query_envelope, poll_messages, register) for new coverage. Raw relay polls return envelopes (payload under body), not response bodies.

Editing the spec

• Read all of rfc.txt before editing; it is the sole source of truth and is deliberately terse.
• Keep the plain-text single-file format. Don't restructure into Markdown files unless asked.
• Invariants I1–I9 (§2) are normative; proposals contradicting them (scores in responses, topic taxonomy, announce stream, dispute messages, replayable broadcast) are out of scope by design.
• Appendix B (Purge Log) is normative: a rejected mechanism may only be re-proposed if the written rationale is addressed.
• §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something.
• Use the spec's vocabulary — member/querier/responder, citations/receipts/aggregates, source/enrichment members — not client/server or search-engine terms.

Implementation notes

• Envelope signing is provisional (src/crypto.rs): FRX/0.3 + fields + sorted-key canonical JSON body. §10's signature canonicalization open issue is unsolved — never present this scheme as interoperable.
• Phase 1 only: no aggregates (Phase 2), no receipts/lineage/delegation (Phase 3). Responses travel relay-mediated unicast; transport is HTTP long-poll, not SSE.
• Relay verifies signatures, carries only query broadcasts, holds no history (queue drained on poll), and returns 429 + Retry-After under backpressure — never silent drops.
• Responder searches only collections marked shared (I9), stays silent when nothing matches, and emits ordered results with honest truncated/more_available and no scores (I6).
• Index layout: Tantivy at <data_dir>/index, collections manifest at <data_dir>/collections.toml; exposure (metadata|full) gates whether content is returned.
• Egress checks live in the responder path (src/node.rs respond), not the relay — keep private collections unreachable there.
• Appendix B is executable policy: tests/purges.rs has one absence test per rejected mechanism (11 rows, not the RFC's informal "nine"). Add a negative test there before ever re-proposing one, and only if the rationale is addressed.
• add/reindex reset a collection (delete by manifest name) before re-adding, so deleted files don't linger; collection identity is its name, and same-named collections replace each other.
• Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale).

Known gaps (Phase 2/3, intentional — don't fake them)

• No aggregate serving/dashboard (RFC §9 conformance is partial without it), no directory watching (new files need reindex), no receipts/lineage/delegation, no TLS, no consumer admission/relay discovery.
• Node query dedup is by qid only; there is no envelope replay/nonce window (RFC doesn't require one).

Technical plans (deliberately not in the RFC)

• Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface.
• Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as enrichment members / backfill seeding. RFC §6 and Appendix A already name both as example derived corpora, so no new mechanisms are required; enrichment members are metadata-only exposure and sit outside the citation market.
• Phase 1 (two-node query/response) is built and tested; the enrichment demo layers on top of it.
• Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate.
• frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher".
• Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role.
• Matching accuracy is a project-health concern: start lexical (Tantivy), plan a hybrid cheap lexical gate + optional local embedding rerank (two-stage ingestion, Appendix A); embedding model stays local and replaceable (I2/I5).