From dfa1fb114343b4111a4b6fcfc8830eb10dd8e805 Mon Sep 17 00:00:00 2001 From: George Coles Date: Tue, 15 Sep 2026 12:59:35 -0400 Subject: [PATCH] Remove membership classes (spec, registry schema, enforcement); no roles or tiers --- AGENTS.md | 10 +++--- DEPLOY.md | 2 +- DESIGN.md | 2 +- rfc.txt | 4 +-- src/commands.rs | 59 +++++++++------------------------- src/config.rs | 9 ------ src/main.rs | 20 +++--------- src/node.rs | 28 ++++++----------- src/registry.rs | 14 +++------ tests/aggregates.rs | 75 ++------------------------------------------ tests/cli.rs | 6 ++-- tests/conformance.rs | 2 -- tests/encryption.rs | 1 - tests/federation.rs | 9 +++--- tests/onboarding.rs | 36 +++++++++------------ tests/registry.rs | 13 +++----- 16 files changed, 68 insertions(+), 222 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index edbe9e3..c062984 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ - Invariants I1–I9 (§2) are normative; proposals contradicting them (scores in responses, topic taxonomy, announce stream, dispute messages, replayable broadcast, in-protocol pricing/settlement) are out of scope by design. - Appendix B (Purge Log) is normative: a rejected mechanism may only be re-proposed if the written rationale is addressed. - §10 Open Issues are known gaps, not oversights (e.g., signature canonicalization blocks Phase-1 interop). Check it before "fixing" something. -- Use the spec's vocabulary — member/querier/responder, aggregates, source/enrichment members — not client/server or search-engine terms. +- Use the spec's vocabulary — member/querier/responder, aggregates — not client/server or search-engine terms. - A new `MUST` is only legitimate if it is observable at the boundary, deterministically verifiable by a peer, beneficial to the counterparty, and not derivable from local policy. Ranking/ordering/presentation fails this test and stays local (§5); scores never travel (I6). - I2 is not blanket anti-centralization: shared coordination (identity, admission, contract) is centralized in the MA because common state is cheaper held once; decisions that consume local information (matching, relevance, sharing, retention) stay local. Off-wire conduct (link handling, retention, gating) is contract, not conformance. @@ -31,7 +31,7 @@ - `add`/`reindex` reset a collection (delete by manifest `name`) before re-adding, so deleted files don't linger; collection identity is its name, and same-named collections replace each other. - Relay backpressure is global: any member's full queue 429s every publisher until drained (visible per §3, but one lagging member can stall the firehose — revisit before scale). - Member authority (Draft 0.5 §6): the MA-signed registry snapshot is authoritative when configured (`[node] registry` = file path or URL, `ma_key` pinned; monotonic version — rollback and forgery close the node; file path is mtime-reloaded, URL is fetched at start + every 60s and cached to `/registry-cache.json`, so outage fails static). Keys carry optional validity windows (`not_before`/`not_after`); rotation = `registry add-key` then `revoke-key`. -- `/members.toml` (name, pubkey, class, `previous` keys, mtime-reloaded) is a dev/local fallback used only when no registry is configured; empty directory without a registry is open bootstrap only when `dev_bootstrap = true` (RFC §6: explicit dev flag). Receivers drop content-bearing responses from enrichment-class senders (metadata-only, §6). +- `/members.toml` (name, pubkey, `previous` keys, mtime-reloaded) is a dev/local fallback used only when no registry is configured; empty directory without a registry is open bootstrap only when `dev_bootstrap = true` (RFC §6: explicit dev flag). - MA tooling: `frxd registry init|add|add-key|revoke-key|remove|list|applications|approve|invite|token|revoke-token|set-relays|show|serve` (signed `registry.json` + `ma-key.hex` in `--dir`); `frxd init --id/--registry/--ma-key`; `frxd key show|rotate`; `member add --previous ` for the fallback path. A node with no `[node] relays` discovers them from the registry snapshot (`doc.relays`). - Aggregate semantics are our implementation choices from a terse spec: requests are `aggregate` envelopes carrying only `period`; replies carry `sent` (broadcasts that month) / `passed` (responses consumed from that member); granularity floor is enforced as YYYY or YYYY-MM only (finer rejected), yearly rolls up months. Revisit with §10 sufficiency review. @@ -43,13 +43,13 @@ ## Technical plans (deliberately not in the RFC) - Record plans here — not as spec edits — when they are implementation/demo choices rather than protocol surface. -- Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as enrichment members / backfill seeding. RFC §6 and Appendix A already name both as example derived corpora, so no new mechanisms are required; enrichment members are metadata-only exposure. -- Phase 1 (two-node query/response) is built and tested; the enrichment demo layers on top of it. +- Demo plan: build a useful end-to-end demo on GDELT and Common Crawl (CC-NEWS; sometimes called "OpenCrawl" in discussion) as ordinary members / backfill seeding; Appendix A names both as example derived corpora. Derived corpora are metadata-only via the member's own `exposure=metadata` collection setting (I9) — there is no registry-level class. +- Phase 1 (two-node query/response) is built and tested; the derived-corpora demo layers on top of it. - Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate. - frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher". - Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role. - Matching floor: boundary tokenizer (`src/tokenizer.rs` — letter/digit splits so `5555` matches `DLEX5555`, lowercase, ASCII fold, English stopwords+stemmer) → coverage gate (`[match] min_coverage`, default 0.4; 1–2 term queries require all terms) → title boost 2.0 + phrase boost 3.0 + query-time snippets. Schema changes require a fresh index dir (`open_or_create` errors on mismatch). - Engine seam: `src/engine.rs` `SearchEngine` trait (`search` → `EngineOutput { hits, total: Option }`, `doc_count`); `respond()` in `src/node.rs` is the conformance wrapper (budget clamp, truncation from engine total — unknown total forces `truncated = true`). Power users can implement the trait (HTTP adapter or subprocess to an external engine). -- Onboarding: `frxd --onboarding` runs a wizard consuming a credential block (`id=.. token=.. registry=.. ma_key=..`) issued by the MA (`registry serve`; HTML page at `/`, `POST /v1/signup` queues a pending application, `POST /v1/enroll` binds keys and re-signs). Identity registration stays MA-side; the wizard never creates identities, only binds locally generated keys. Applications live in `/applications.json` (mode 600, MA contract data — never in the signed snapshot); `frxd registry approve ` promotes one (member stub + credential block whose token is the member's reusable account credential, hashed in `/tokens.json` — authorizes key enrollment for every node the member runs); applicants never self-declare a class — the MA assigns it with `approve --class enrichment` (default source; classes are provenance, not roles — every member may query and respond, I5). `frxd registry token ` mints another member token, `revoke-token ` revokes all of a member's tokens; `frxd registry invite ` mints a single-use 24h handoff token (`/invites.json`). Prompts accept empty input as the default; scripted stdin works for tests. +- Onboarding: `frxd --onboarding` runs a wizard consuming a credential block (`id=.. token=.. registry=.. ma_key=..`) issued by the MA (`registry serve`; HTML page at `/`, `POST /v1/signup` queues a pending application, `POST /v1/enroll` binds keys and re-signs). Identity registration stays MA-side; the wizard never creates identities, only binds locally generated keys. Applications live in `/applications.json` (mode 600, MA contract data — never in the signed snapshot); `frxd registry approve ` promotes one (member stub + credential block whose token is the member's reusable account credential, hashed in `/tokens.json` — authorizes key enrollment for every node the member runs). `frxd registry token ` mints another member token, `revoke-token ` revokes all of a member's tokens; `frxd registry invite ` mints a single-use 24h handoff token (`/invites.json`). Prompts accept empty input as the default; scripted stdin works for tests. - Next matching steps: eval harness with a small golden set (precision@k + false-silence rate), then a dense recall leg (model2vec-rs 0.2.1 exists but needs `default-features = false, features = ["fancy-regex", "local-only"]` for musl/airgapped; verify crate + model licenses before bundling), then an optional cross-encoder reranker. Embeddings are for recall; reranking is the precision tier. - Identity/registry (RFC Draft 0.5 §4/§6): MA-hosted FQDN identifiers first (`