Compare commits

12 Commits
27 changed files with 1949 additions and 56 deletions
+3
View File
@@ -1 +1,4 @@
/target /target
/frx-data/
/comments.txt
/relay.log
+7 -2
View File
@@ -2,8 +2,10 @@
## Repo shape ## Repo shape
- `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries). - `rfc.txt` (FRX — Federated Retrieval Exchange, Draft 0.5) is the normative spec; `src/` is the Phase 1 `frxd` implementation (single crate, two binaries).
- `DESIGN.md` is the partner-facing architecture and decision log (with rationale for the spec cuts); keep it in sync when architecture decisions change.
- `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands. - `frxd` is the member node (init/add/index/serve/relay/query/status); `frx` is the thin client (search/query/status). Relay and node roles are separate subcommands.
- Commands: `cargo build`, `cargo test` (90 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config. - `DEPLOY.md` documents the TLS/deployment story: members need no TLS (outbound HTTPS), relays terminate TLS with Caddy or a tunnel, `ca_cert` adds private CAs, `allow_insecure` opts into plain http on private networks, and non-loopback `http://` is refused by default.
- Commands: `cargo build`, `cargo test` (104 tests: unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config. unit in `src/`; e2e `tests/phase1.rs`; conformance `tests/conformance.rs`; aggregates + member directory `tests/aggregates.rs`; registry `tests/registry.rs`; federation/isolation/admission `tests/federation.rs`; SSE `tests/streaming.rs`; encrypted unicast `tests/encryption.rs`; concurrency/restart `tests/concurrency.rs`; real subprocess CLI `tests/cli.rs`; 1000-doc `tests/scale.rs`; purge-log absence `tests/purges.rs`; shared fixtures `tests/common/mod.rs`). No CI/lint config.
- E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies. - E2E pattern: relay + nodes in-process on ephemeral ports with tempdir corpora; use `tests/common/mod.rs` helpers (`spawn_relay*`, `query_envelope`, `poll_messages`, `register`) for new coverage. Raw relay polls return envelopes (payload under `body`), not response bodies.
## Editing the spec ## Editing the spec
@@ -46,5 +48,8 @@
- Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate. - Language: Rust (settled, matches §7). Decided by the engine requirement, not preference: Tantivy gives in-process Lucene-class BM25 + incremental indexing; C/C++ embedded alternatives are worse (Xapian GPL-2+, CLucene unmaintained, SQLite FTS5 thin), plus single static musl binaries for the install story and memory safety on the untrusted network/crypto path. Don't re-litigate.
- frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher". - frxd modes (one binary, config toggles, no code required of publishers): querier (broadcast/local-first search), responder (match incoming queries against shared collections, sign), local index (watch dirs, extract text, explicit shared marking per I9). Use RFC terms querier/responder, not "subscriber/publisher".
- Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role. - Roles are not exclusive: a single node may issue queries and answer them concurrently (I5, §3 "any member"). Implement querier/responder as independent enable flags — never an exclusive mode enum or fixed deployment role.
- Matching accuracy is a project-health concern: start lexical (Tantivy), plan a hybrid cheap lexical gate + optional local embedding rerank (two-stage ingestion, Appendix A); embedding model stays local and replaceable (I2/I5). - Matching floor: boundary tokenizer (`src/tokenizer.rs` — letter/digit splits so `5555` matches `DLEX5555`, lowercase, ASCII fold, English stopwords+stemmer) → coverage gate (`[match] min_coverage`, default 0.4; 12 term queries require all terms) → title boost 2.0 + phrase boost 3.0 + query-time snippets. Schema changes require a fresh index dir (`open_or_create` errors on mismatch).
- Engine seam: `src/engine.rs` `SearchEngine` trait (`search``EngineOutput { hits, total: Option<u64> }`, `doc_count`); `respond()` in `src/node.rs` is the conformance wrapper (budget clamp, truncation from engine total — unknown total forces `truncated = true`). Power users can implement the trait (HTTP adapter or subprocess to an external engine).
- Onboarding: `frxd --onboarding` runs a wizard consuming a credential block (`id=.. token=.. registry=.. ma_key=..`) issued by the MA's signup endpoint (`registry serve --signup-code --registry-url`; HTML page at `/`, `POST /v1/signup` → one-time invite token, `POST /v1/enroll` binds keys and re-signs). Identity registration stays MA-side; the wizard never creates identities, only binds locally generated keys. Invites live in `<registry dir>/invites.json`. Prompts accept empty input as the default; scripted stdin works for tests.
- Next matching steps: eval harness with a small golden set (precision@k + false-silence rate), then a dense recall leg (model2vec-rs 0.2.1 exists but needs `default-features = false, features = ["fancy-regex", "local-only"]` for musl/airgapped; verify crate + model licenses before bundling), then an optional cross-encoder reranker. Embeddings are for recall; reranking is the precision tier.
- Identity/registry (RFC Draft 0.5 §4/§6): MA-hosted FQDN identifiers first (`<label>.frx.<ma-domain>`, no DNS needed by users), signed versioned registry snapshot with the MA key pinned; envelope `from` = identifier, `key` = pubkey; registry outage fails static. Member-hosted identities, MA anchor rollover, and unicast confidentiality are §10 open. Implementation phases: A (signed registry snapshot) and B (identifier + `key` + JCS on the wire) are built and tested. Prioritize frictionless onboarding (users may be department-level and cannot create DNS). - Identity/registry (RFC Draft 0.5 §4/§6): MA-hosted FQDN identifiers first (`<label>.frx.<ma-domain>`, no DNS needed by users), signed versioned registry snapshot with the MA key pinned; envelope `from` = identifier, `key` = pubkey; registry outage fails static. Member-hosted identities, MA anchor rollover, and unicast confidentiality are §10 open. Implementation phases: A (signed registry snapshot) and B (identifier + `key` + JCS on the wire) are built and tested. Prioritize frictionless onboarding (users may be department-level and cannot create DNS).
+5
View File
@@ -24,6 +24,11 @@ tokio = { version = "1.53.1", features = ["full"] }
toml = "1.1.6" toml = "1.1.6"
x25519-dalek = { version = "2", features = ["static_secrets"] } x25519-dalek = { version = "2", features = ["static_secrets"] }
[profile.release]
lto = true
codegen-units = 1
strip = true
[dev-dependencies] [dev-dependencies]
bytes = "1.12.1" bytes = "1.12.1"
tempfile = "3.27.0" tempfile = "3.27.0"
+215
View File
@@ -0,0 +1,215 @@
# Deploying FRX
Three roles: **member node** (`frxd serve`), **relay** (`frxd relay`), **registry** (`frxd registry`, run by the MA).
TLS is only a concern for servers. Members connect outbound and need no domain, port, or certificate.
## Member node (zero TLS work)
```
frxd init --name alice --id alice.frx.federatedsearch.org \
--registry https://ma.federatedsearch.org/registry.json --ma-key <ma-hex> \
--relay https://relay.federatedsearch.org --data-dir ./alice-data
frxd add ~/documents --name docs --shared --exposure full
frxd serve
```
The node connects outbound over HTTPS, verifies the registry with the pinned MA key, and holds an SSE stream per relay. Nothing inbound, no DNS, no certificates. Members at departmental level can start here.
## Joining the network (membership site + wizard)
The apex (`federatedsearch.org`) proxies the same membership page; `ma.` is the service host.
Identity registration stays with the MA; the wizard only consumes the credentials it issues.
MA operator — run the signup site:
```
frxd registry --dir ./ma init --zone frx.federatedsearch.org
frxd registry --dir ./ma serve --listen 127.0.0.1:7800 --signup-code <code> --registry-url https://ma.federatedsearch.org/registry.json
```
(put Caddy in front for a real domain). The page at `/` accepts a label and the signup code and returns a credential block: `id=... token=... registry=... ma_key=...`.
New member:
```
frxd --onboarding
```
The wizard asks for a config path, accepts the pasted credential block (or field-by-field entry), generates keypairs, enrolls the new key with the MA (redeeming the single-use invite token), verifies the signed registry snapshot, takes relay defaults from the registry, optionally shares a directory, writes the config, and offers to start serving. If enrollment is unavailable (file-path registry), it prints the exact `registry add` command the operator must run.
## Relay with TLS (one line)
Run `frxd` on loopback and terminate TLS with Caddy:
```
caddy reverse-proxy --from relay.federatedsearch.org --to 127.0.0.1:7700
```
Caddyfile equivalent (apex is the public front door, `ma.` the membership service, `relay.` the relay, `git.` the code host — all on one host):
```
federatedsearch.org {
reverse_proxy 127.0.0.1:7800
}
ma.federatedsearch.org {
reverse_proxy 127.0.0.1:7800
}
relay.federatedsearch.org {
reverse_proxy 127.0.0.1:7700
}
git.federatedsearch.org {
reverse_proxy 127.0.0.1:3000
}
```
Relay command (peers and registry gated by the MA):
```
frxd relay --listen 127.0.0.1:7700 \
--url https://relay.federatedsearch.org \
--peer https://relay2.federatedsearch.org \
--registry https://ma.federatedsearch.org/registry.json --ma-key <ma-hex>
```
No domain or open ports? Tunnel it:
```
frxd relay --listen 127.0.0.1:7700 --allow-insecure
cloudflared tunnel --url http://127.0.0.1:7700
```
`systemd` unit example:
```ini
[Unit]
Description=FRX relay
After=network-online.target
[Service]
ExecStart=/usr/local/bin/frxd relay --listen 127.0.0.1:7700 \
--url https://relay.federatedsearch.org \
--registry https://ma.federatedsearch.org/registry.json --ma-key <ma-hex>
Restart=on-failure
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
[Install]
WantedBy=multi-user.target
```
The same unit shape works for `frxd serve` (add `--config /etc/frxd/frxd.toml`).
## Registry (MA)
```
frxd registry --dir /var/lib/frxd/registry init
frxd registry --dir /var/lib/frxd/registry add alice.frx.federatedsearch.org <key> --enc-key <enc>
frxd registry --dir /var/lib/frxd/registry serve --listen 127.0.0.1:7800
```
Put the same Caddy in front, or distribute `registry.json` out of band (it is signed, so the channel does not matter). The snapshot is versioned; nodes reject rollback and fail static during outages.
## Code hosting and downloads (Gitea)
The public source and release binaries live at `git.federatedsearch.org` (Gitea), so the
download step on the membership page stays on infrastructure the federation operates.
One-time install on the server (root):
```
VER=1.27.3
curl -fsSLO https://dl.gitea.com/gitea/$VER/gitea-$VER-linux-amd64{,.sha256}
sha256sum -c gitea-$VER-linux-amd64.sha256
install -m 0755 gitea-$VER-linux-amd64 /usr/local/bin/gitea
adduser --system --shell /bin/bash --gecos 'Gitea' --home /home/git --group git
mkdir -p /var/lib/gitea/{custom,data,log} /etc/gitea && chown -R git:git /var/lib/gitea /etc/gitea
```
`/etc/gitea/app.ini` essentials (rest defaults; secrets via `gitea generate secret`):
```ini
WORK_PATH = /var/lib/gitea
[database]
DB_TYPE = sqlite3
PATH = /var/lib/gitea/data/gitea.db
[server]
DOMAIN = git.federatedsearch.org
SSH_DOMAIN = git.federatedsearch.org
ROOT_URL = https://git.federatedsearch.org/
HTTP_ADDR = 127.0.0.1
HTTP_PORT = 3000
[security]
INSTALL_LOCK = true
[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = false
```
systemd unit (`User=git`, `ExecStart=/usr/local/bin/gitea web --config /etc/gitea/app.ini`,
`WorkingDirectory=/var/lib/gitea`), then `gitea migrate --config /etc/gitea/app.ini` as the
`git` user and `systemctl enable --now gitea`. Git-over-SSH uses the host sshd via the `git`
user's Gitea-managed `authorized_keys`; HTTPS pushes can use an access token instead.
Admin bootstrap (as `git` user):
```
gitea admin user create --admin --username <you> --email <you>@federatedsearch.org --random-password --config /etc/gitea/app.ini
gitea admin user generate-access-token -u <you> -t bootstrap --scopes all --config /etc/gitea/app.ini
```
The org is `frx`, the repo `frxd` → clone URL
`https://git.federatedsearch.org/frx/frxd.git`. Membership stays closed (signup code);
repo reads are public.
Publishing a release (from the checkout):
```
git tag v0.1.0 && git push gitea v0.1.0
# build static binaries (see next section), then attach via the API:
curl -X POST https://git.federatedsearch.org/api/v1/repos/frx/frxd/releases \
-H "Authorization: token <token>" -H 'content-type: application/json' \
-d '{"tag_name":"v0.1.0","name":"v0.1.0"}'
curl -X POST https://git.federatedsearch.org/api/v1/repos/frx/frxd/releases/<id>/assets?name=frxd-linux-amd64 \
-H "Authorization: token <token>" -F attachment=@frxd-linux-amd64
```
Asset URLs follow `/frx/frxd/releases/download/<tag>/<file>` — the membership page pins
those. Bump the page when a release changes.
## Private networks and custom CAs
- `ca_cert = "/etc/ssl/private-ca.pem"` in `[node]`, or `--ca-cert` on the relay: adds a private/corporate root CA for relay and registry connections.
- `allow_insecure = true` / `--allow-insecure`: explicit opt-in for plain `http://` on a VPN/LAN. Without it, non-loopback `http://` endpoints are refused at startup.
- `http://127.0.0.1` is always allowed for development.
## Static binary
```
rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl
```
`[profile.release]` enables LTO and stripping. All dependencies are pure Rust, so the musl build has no system-library requirements. `ring` needs a musl C toolchain (`musl-tools`); without host sudo, build in a container instead:
```
docker run --rm -v "$PWD":/src -w /src rust:1-slim-bookworm bash -c \
"apt-get update -qq && apt-get install -y -qq musl-tools && rustup target add x86_64-unknown-linux-musl && cargo build --release --target x86_64-unknown-linux-musl"
```
Binaries land in `target/x86_64-unknown-linux-musl/release/`; rename to
`frxd-linux-amd64` / `frx-linux-amd64` for release assets, with `sha256sum` sidecar files.
## What TLS does and does not cover
Envelopes are signed and registry snapshots are MA-signed, so TLS is not what protects message authenticity or registry integrity. TLS protects traffic from network observers, authenticates the relay endpoint, and hides mailbox metadata. Unicast response bodies are already encrypted end-to-end to the recipient's X25519 key.
+137
View File
@@ -0,0 +1,137 @@
# FRX — Design and Decisions
Status: Draft 0.5 (experimental), reference implementation `frxd` in Rust.
This document is the architecture and decision record. The normative protocol surface is `rfc.txt`; deployment is `DEPLOY.md`; agent-facing notes are `AGENTS.md`.
## 1. What FRX is
FRX is a membership federation for retrieval. Members answer broadcast queries from content they already hold; there is no supply announcement stream and no in-protocol payment. The protocol standardizes the message layer and honesty constraints only: signed envelopes, budgets, honest truncation, egress consent, aggregate courtesy. Matching, relevance, ranking, retention, and trust are local.
Any member may originate queries and answer them; roles are enable flags, never a deployment role (§I5).
## 2. Components
- **Member node (`frxd serve`)** — owns a keypair and an identifier, indexes local collections, broadcasts queries, answers queries from shared collections, receives responses. Local-first: local results are merged with remote results, provenance-marked.
- **Relay (`frxd relay`)** — dumb, interchangeable transport. Holds no history, replays nothing, fans queries out to subscribed members, carries unicast responses/aggregates to member mailboxes. Relays may peer with each other to flood queries.
- **Registry (MA)** — the membership authority: a signed, versioned snapshot listing identifiers, classes, authorized keys with validity windows, optional X25519 encryption keys, and relay endpoints. The registry is the sole authority for key-to-identifier binding.
## 3. Message flow
```
querier --publish(signed query)--> relay A --flood--> relay B
| |
mailbox fanout to all subscribed members
| |
responder (on B) matches shared collections, signs response, encrypts to querier
responder --unicast(signed ciphertext)--> relay network --> querier mailbox
```
- Queries are broadcast live to all members; silence is conformant and informative.
- Responses are unicast to the querier and addressed by transport key.
- Aggregates are bilateral, on request, per member, per period.
- Members hold one authenticated SSE stream per relay (`/v1/stream`), with long-poll fallback on 404/405.
- Relay federation is copy-only, hop-bounded, and duplicate-suppressed by envelope signature; direct publishes are never suppressed.
## 4. Identity and trust
- **Identifier**: an MA-hosted FQDN (`alice.frx.federatedsearch.org`). No member-controlled DNS is required. Member-hosted identifiers (keys published in the member's own DNS, allowlisted by the MA) are planned, not normative.
- **Credentials**: keys are rotatable and carry validity windows; multiple keys may be valid during rotation. Rotation publishes a successor before retiring the predecessor; revocation removes a key or shortens validity. A key never extends its own authority.
- **Registry trust**: nodes pin the MA key. Snapshots are versioned (rollback rejected), signature-verified, and cached; registry outage fails static on the last validated snapshot. Open bootstrap requires an explicit development flag.
- **Envelope authentication**: `{type, from, key, ts, nonce, body, sig}`; the signature covers the JCS (RFC 8785) canonical form of the unsigned envelope under a versioned prefix, and the receiver verifies both the signature and the registry binding `map[key].id == from`. Golden bytes and a deterministic signature are pinned in `tests/conformance.rs`.
- **Freshness**: envelopes outside ±300 s are rejected. A node-side nonce cache is not implemented (replay inside the window is possible).
## 5. Security posture
What protects what:
| Concern | Mechanism |
| --- | --- |
| Message authenticity | Ed25519 signature over the JCS envelope |
| Key-to-identifier binding | MA-signed registry snapshot, pinned anchor |
| Registry freshness | Monotonic version, fail-static cache |
| Mailbox access | Challenge-response proof of key possession, single-use nonce |
| Transport observation | TLS at relays (reverse proxy or tunnel), optional private CA |
| Unicast confidentiality | X25519 / HKDF-SHA256 / ChaCha20-Poly1305 to the recipient's registry key |
| Query confidentiality | None by design: receiver-local matching needs plaintext at members |
Accepted limitations: relays see queries in clear by design (I3 constrains what may enter broadcasts); no end-to-end encryption is possible for queries; no directory-free admission on relays (optional gate); no bilateral node-side rate limiting yet; the local control API is unauthenticated and must stay on loopback.
## 6. Economic and governance stance
- Economics is out of protocol scope. The protocol carries no pricing, metering, settlement, citations, or receipts. Payments, licensing, and content transactions happen at the edge, on the owner's terms (e.g., `exposure: metadata` keeps content behind the owner's endpoint).
- The MA governs identity, admission, and contract — who, never quality. Expulsion grounds are fabrication, admission fraud, and sustained abuse.
- Off-wire conduct (link handling, retention, gating) is contractual; the protocol neither observes nor adjudicates it.
- Aggregates advise only; they are inadmissible as sanction evidence. Defaults and relay governance (§10) remain the main soft-centralization risks.
## 7. Decision log
Decisions taken during design review, with rationale.
| # | Decision | Rationale | Status |
| --- | --- | --- | --- |
| 1 | Minimal normative surface; all judgment local | Interop only needs the message layer; ranking/trust are local information problems | RFC §1–§2, §5 |
| 2 | Role symmetry; querier/responder are flags | No privileged roles; one node may both ask and answer (I5) | Implemented |
| 3 | Pull-only supply; no announce stream | A supply firehose adds cost and privacy exposure; queries already reach all members (I7, App. B) | RFC, implemented |
| 4 | No citation/receipt economics | A retrieval protocol cannot observe citations on the web; self-issued artifacts have no trust anchor; removed in Draft 0.5 (App. B row) | Removed |
| 5 | No scores on the wire; ordering/presentation local | Any ordering MUST is unfalsifiable without a standard scorer; scores invite open-ended comparability and reputation machinery | I6, §5 |
| 6 | Eager/lazy retention removed from the spec | Content housekeeping is unobservable between peers; not an invariant | I5/§5 cleaned |
| 7 | Honest truncation, not result-count etiquette | Quantity is querier-local, selection responder-local; budget plus a truncation flag suffice | §4, App. B |
| 8 | Relay-mediated fanout with peer flooding; per-member isolation | O(1) publish; relays stay dumb and interchangeable; one lagging member must not stall the firehose | §3, implemented |
| 9 | Visible backpressure, never silent drops | Lagging members get 429 + `missed` or an SSE `lag` event; publishers are unaffected | §3/§9, implemented |
| 10 | Stable identifier + rotatable credentials; signed registry | Identity survives rotation; admission is gated once, credentials are self-managed; outage fails static | §4/§6, implemented |
| 11 | `from` = identifier, `key` = pubkey; JCS canonical form | Resolves §10 canonicalization with a cross-language standard and a pinned golden vector | Implemented |
| 12 | Relays address mailboxes by key, not by identifier | Keeps relays ignorant of identity and makes rotation local | Implemented |
| 13 | SSE first, long-poll fallback | Push latency and connection efficiency; fallback for restricted networks | Implemented |
| 14 | Unicast confidentiality profile (X25519/HKDF/ChaCha20-Poly1305) | Relays carry ciphertext; queries cannot be private (broadcast plus local matching) | Implemented, not yet normative |
| 15 | Centralize coordination, localize judgment (I2) | Common state is cheaper held once: identity, admission, contract in the MA; matching, relevance, sharing, retention local | I2 reframed |
| 16 | No sessions; per-message signatures | Peers are not connected; mailbox auth is a transport-local proof of possession | Implemented |
| 17 | Lexical coverage gate before any rerank | Precision is project health; a demo false positive showed raw OR matching is too weak; embeddings later, local and replaceable | Implemented (`[match] min_coverage`) |
| 18 | Default engine: boundary tokenizer + fold + stopwords + stemmer, title/phrase boosts, query-time snippets | The floor must be high out of the box; model-number and morphology matching are cheap wins with no model | Implemented (`src/tokenizer.rs`) |
| 19 | Engine seam: `SearchEngine` trait with the responder path as the conformance wrapper | Plugins can change quality, never conformance; engines return `Option<total>` so an external engine can't fake the truncation bit | Implemented (`src/engine.rs`) |
| 20 | Onboarding is a wizard consuming MA-issued credentials; signup lives on the MA's site | Users may be department-level and cannot create domains or DNS records; the wizard never creates identities, only binds locally generated keys | Implemented (`frxd --onboarding`, `/v1/signup`+`/v1/enroll`) |
## 8. Implementation status
Built and tested (99 tests):
- Envelope, JCS signing, registry binding, freshness window
- Tantivy index, collections manifest, shared/exposure enforcement, reindex reset
- Coverage-gated lexical matching with boundary tokenization, stemming, folding, boosts, and snippets
- Engine seam (`SearchEngine` trait) behind the conformance wrapper
- Query broadcast, SSE streaming, long-poll fallback, per-member queues and lag reporting
- Relay federation, relay admission, registry watcher (path/URL, monotonic, fail-static)
- Registry CLI (init/add/add-key/revoke-key/set-enc-key/set-relays/show/serve), key rotation
- Aggregates (sent/passed, monthly floor, yearly rollup)
- Encrypted unicast, TLS guardrails and custom CA support, static-build release profile
- Onboarding wizard (`frxd --onboarding`) and MA signup/enroll site (`/v1/signup`, `/v1/enroll`, HTML at `/`)
Not built (see §9): dashboard UI, directory watching, user-supplied URL ingestion, node-side rate limiting, member-hosted identities, delegation, document lineage, MA anchor rollover, embedding rerank, invite-based self-enrollment, relay-to-relay unicast routing.
## 9. Open issues
From RFC §10 and implementation findings:
- **Response routing across relays** — unicast is delivered on the relay where the recipient is subscribed; a responder whose configured relays do not include the recipient's relay cannot deliver. Current workaround: members connect to multiple relays. Relay-to-relay unicast forwarding is not implemented.
- **Consumer admission tier** — automated/invite admission without weakening the Sybil defense.
- **Default-relay governance** — registry-listed relays settle discovery; who operates the defaults remains a soft centralization point.
- **Member-hosted identifiers** — keys in the member's own DNS instead of the MA registry.
- **MA anchor rollover** — successor commitment and overlap for the registry signing key.
- **Delegation** — granting authority to agents/sub-identities; unspecified.
- **Document lineage** — revision/supersedes without a supply stream; unspecified.
- **Unicast confidentiality profile** — implemented but not normative.
- **Aggregate semantics** — counter definitions and the granularity floor are implemented choices from a terse spec; revisit with the sufficiency review.
- **Replay** — ±300 s window only; no node nonce cache.
- **Matching** — thresholds are untuned pending a real corpus; embedding rerank optional and local.
## 10. Glossary
- **Member** — an entity holding a keypair and a registry-listed identifier.
- **Querier / responder** — the asking and answering roles of any member.
- **Relay** — dumb transport that fans out broadcasts and holds member mailboxes.
- **Registry / MA** — the membership authority and its signed snapshot.
- **Envelope** — the signed message framing shared by all message types.
- **Broadcast** — a query delivered to every subscribed member.
- **Unicast** — a response or aggregate addressed to one member.
- **Aggregate** — courtesy counters served bilaterally on request.
- **Collection / shared / exposure** — local index unit; egress consent flag; metadata vs full content release.
- **Coverage gate** — the minimum fraction of query terms a document must match to be a candidate.
+358 -8
View File
@@ -5,8 +5,9 @@ use anyhow::{Context, Result, anyhow};
use axum::extract::State; use axum::extract::State;
use axum::http::StatusCode; use axum::http::StatusCode;
use axum::response::{IntoResponse, Response}; use axum::response::{IntoResponse, Response};
use axum::routing::get; use axum::routing::{get, post};
use axum::{Json, Router}; use axum::{Json, Router};
use serde::Deserialize;
use tokio::net::TcpListener; use tokio::net::TcpListener;
use crate::config::{CLASS_ENRICHMENT, CLASS_SOURCE, Config, Member, load_members, save_members}; use crate::config::{CLASS_ENRICHMENT, CLASS_SOURCE, Config, Member, load_members, save_members};
@@ -272,7 +273,7 @@ fn mutate_registry(dir: &Path, mutate: impl FnOnce(&mut RegistryDoc) -> Result<(
Ok(signed.doc.version) Ok(signed.doc.version)
} }
pub fn registry_init(dir: &Path) -> Result<()> { pub fn registry_init(dir: &Path, zone: &str) -> Result<()> {
let registry_path = registry_doc_path(dir); let registry_path = registry_doc_path(dir);
if registry_path.exists() { if registry_path.exists() {
return Err(anyhow!( return Err(anyhow!(
@@ -288,6 +289,7 @@ pub fn registry_init(dir: &Path) -> Result<()> {
version: 1, version: 1,
issued_at: now_ts(), issued_at: now_ts(),
ma_key: String::new(), ma_key: String::new(),
zone: zone.to_string(),
members: Vec::new(), members: Vec::new(),
relays: Vec::new(), relays: Vec::new(),
}; };
@@ -442,12 +444,38 @@ pub fn registry_show(dir: &Path) -> Result<()> {
Ok(()) Ok(())
} }
pub async fn registry_serve(dir: &Path, listen: &str) -> Result<()> { struct RegistryServer {
let state = dir.to_path_buf(); dir: PathBuf,
let app = Router::new() signup_code: Option<String>,
registry_url: Option<String>,
}
pub fn registry_router(
dir: &Path,
signup_code: Option<String>,
registry_url: Option<String>,
) -> Router {
let state = std::sync::Arc::new(RegistryServer {
dir: dir.to_path_buf(),
signup_code,
registry_url,
});
Router::new()
.route("/health", get(registry_health)) .route("/health", get(registry_health))
.route("/registry.json", get(registry_snapshot)) .route("/registry.json", get(registry_snapshot))
.with_state(state); .route("/", get(registry_page))
.route("/v1/signup", post(registry_signup))
.route("/v1/enroll", post(registry_enroll))
.with_state(state)
}
pub async fn registry_serve(
dir: &Path,
listen: &str,
signup_code: Option<String>,
registry_url: Option<String>,
) -> Result<()> {
let app = registry_router(dir, signup_code, registry_url);
let listener = TcpListener::bind(listen).await?; let listener = TcpListener::bind(listen).await?;
println!("registry serving on http://{}", listener.local_addr()?); println!("registry serving on http://{}", listener.local_addr()?);
axum::serve(listener, app).await?; axum::serve(listener, app).await?;
@@ -458,8 +486,8 @@ async fn registry_health() -> &'static str {
"ok" "ok"
} }
async fn registry_snapshot(State(dir): State<PathBuf>) -> Response { async fn registry_snapshot(State(server): State<std::sync::Arc<RegistryServer>>) -> Response {
match registry::load_registry(&registry_doc_path(&dir)) { match registry::load_registry(&registry_doc_path(&server.dir)) {
Ok(signed) => Json(signed).into_response(), Ok(signed) => Json(signed).into_response(),
Err(_) => ( Err(_) => (
StatusCode::NOT_FOUND, StatusCode::NOT_FOUND,
@@ -469,6 +497,328 @@ async fn registry_snapshot(State(dir): State<PathBuf>) -> Response {
} }
} }
fn sanitize_label(input: &str) -> String {
let mut label = String::new();
let mut last_dash = true;
for c in input.to_ascii_lowercase().chars() {
if c.is_ascii_alphanumeric() {
label.push(c);
last_dash = false;
} else if !last_dash && (c.is_whitespace() || c == '-' || c == '_' || c == '.') {
label.push('-');
last_dash = true;
}
if label.len() >= 32 {
break;
}
}
label.trim_matches('-').chars().take(32).collect()
}
#[derive(Deserialize)]
struct SignupRequest {
label: String,
code: Option<String>,
}
async fn registry_signup(
State(server): State<std::sync::Arc<RegistryServer>>,
Json(request): Json<SignupRequest>,
) -> Response {
let Some(expected) = &server.signup_code else {
return (
StatusCode::FORBIDDEN,
Json(serde_json::json!({ "error": "signup is not enabled" })),
)
.into_response();
};
if request.code.as_deref() != Some(expected.as_str()) {
return (
StatusCode::FORBIDDEN,
Json(serde_json::json!({ "error": "wrong signup code" })),
)
.into_response();
}
let label = sanitize_label(&request.label);
if label.is_empty() {
return (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({ "error": "label must be alphanumeric" })),
)
.into_response();
}
let signed = match registry::load_registry(&registry_doc_path(&server.dir)) {
Ok(signed) => signed,
Err(error) => {
return (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response();
}
};
let zone = signed.doc.zone.clone();
let id = format!("{label}.{zone}");
if signed.doc.members.iter().any(|member| member.id == id) {
return (
StatusCode::CONFLICT,
Json(serde_json::json!({ "error": "member id already taken" })),
)
.into_response();
}
let invite = match registry::create_invite(&server.dir, &id, 24 * 3600) {
Ok(invite) => invite,
Err(error) => {
return (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response();
}
};
if let Err(error) = mutate_registry(&server.dir, |doc| {
doc.members.push(RegistryMember {
id: id.clone(),
class: crate::config::CLASS_SOURCE.to_string(),
keys: Vec::new(),
enc_key: None,
});
Ok(())
}) {
return (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response();
}
let registry_url = server.registry_url.clone().unwrap_or_default();
let ma_key = signed.doc.ma_key.clone();
(
StatusCode::OK,
Json(serde_json::json!({
"id": id,
"token": invite.token,
"registry": registry_url,
"ma_key": ma_key,
"credentials": format!(
"id={id} token={} registry={registry_url} ma_key={ma_key}",
invite.token
),
})),
)
.into_response()
}
#[derive(Deserialize)]
struct EnrollRequest {
id: String,
token: String,
pubkey: String,
enc_key: Option<String>,
}
async fn registry_enroll(
State(server): State<std::sync::Arc<RegistryServer>>,
Json(request): Json<EnrollRequest>,
) -> Response {
let pubkey = match normalize_key(&request.pubkey) {
Ok(pubkey) => pubkey,
Err(error) => {
return (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response();
}
};
let enc_key = match request.enc_key.as_deref().map(normalize_key).transpose() {
Ok(enc_key) => enc_key,
Err(error) => {
return (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response();
}
};
if let Err(error) = registry::redeem_invite(&server.dir, &request.id, &request.token) {
return (
StatusCode::FORBIDDEN,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response();
}
let result = mutate_registry(&server.dir, |doc| {
let Some(member) = doc
.members
.iter_mut()
.find(|member| member.id == request.id)
else {
return Err(anyhow!("unknown member id {}", request.id));
};
if member.keys.iter().any(|entry| entry.key == pubkey) {
return Err(anyhow!("key already authorized"));
}
member.keys.push(KeyEntry {
key: pubkey.clone(),
not_before: now_ts(),
not_after: None,
});
if let Some(enc_key) = enc_key.clone() {
member.enc_key = Some(enc_key);
}
Ok(())
});
match result {
Ok(version) => (
StatusCode::OK,
Json(serde_json::json!({ "id": request.id, "version": version })),
)
.into_response(),
Err(error) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({ "error": error.to_string() })),
)
.into_response(),
}
}
async fn registry_page() -> Response {
(
StatusCode::OK,
[("content-type", "text/html; charset=utf-8")],
REGISTRY_PAGE,
)
.into_response()
}
const REGISTRY_PAGE: &str = r##"<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>FRX — Federated Retrieval Exchange</title>
<style>
* { box-sizing: border-box; }
body { font-family: ui-sans-serif, system-ui, -apple-system, sans-serif; line-height: 1.55;
max-width: 44rem; margin: 0 auto; padding: 2.5rem 1rem; color: #1c1c1e; background: #f7f7f9; }
h1 { font-size: 1.7rem; margin-bottom: 0.2rem; }
h2 { font-size: 1.05rem; margin-top: 2.2rem; }
.tag { color: #555; margin-top: 0; }
.card { background: #fff; border: 1px solid #ddd; border-radius: 10px; padding: 1.1rem 1.25rem; }
code, pre { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 0.92em; }
input, button { font: inherit; border: 1px solid #bbb; border-radius: 6px; padding: 0.45rem 0.6rem; }
input { width: 100%; margin: 0.2rem 0 0.9rem; }
button { background: #174ea6; color: #fff; border: none; cursor: pointer; padding: 0.5rem 1rem; border-radius: 6px; }
button:hover { background: #0f3d91; }
#out { display: none; background: #101418; color: #d6f5d6; padding: 0.85rem 1rem;
border-radius: 8px; white-space: pre-wrap; word-break: break-all; margin-top: 1rem; }
pre.cmd { background: #101418; color: #d6f5d6; padding: 0.85rem 1rem;
border-radius: 8px; overflow-x: auto; }
.muted { color: #666; font-size: 0.92rem; }
ol { padding-left: 1.3rem; }
li { margin: 0.35rem 0; }
a { color: #174ea6; }
</style>
</head>
<body>
<h1>FRX</h1>
<p class="tag">Federated Retrieval Exchange — a membership federation for retrieval.</p>
<p>Members answer broadcast queries from content they already hold. The protocol is deliberately
small: signed messages, budgets, honest truncation, aggregate courtesy. No announce stream, no
scores on the wire, no in-protocol payment.</p>
<p>Everything else — matching, ranking, retention, trust — is local.</p>
<h2>1. Register</h2>
<div class="card">
<form id="f">
<label>Organization or handle<br>
<input name="label" required pattern="[A-Za-z0-9 -]+" placeholder="acme-docs"></label><br>
<label>Signup code (issued by the membership authority)<br>
<input name="code" type="password" placeholder="signup code"></label><br>
<button type="submit">Request membership</button>
</form>
<pre id="out"></pre>
<p class="muted">Your identifier is <code>&lt;label&gt;.frx.federatedsearch.org</code> — no domain or DNS of
your own is needed. Registration returns a one-time credential block:
<code>id=... token=... registry=... ma_key=...</code>.</p>
</div>
<h2>2. Download</h2>
<div class="card">
<p>Static Linux x86_64 binaries (musl — no runtime dependencies):</p>
<pre class="cmd">curl -LO https://git.federatedsearch.org/frx/frxd/releases/download/v0.1.0/frxd-linux-amd64
curl -LO https://git.federatedsearch.org/frx/frxd/releases/download/v0.1.0/frxd-linux-amd64.sha256
curl -LO https://git.federatedsearch.org/frx/frxd/releases/download/v0.1.0/frx-linux-amd64
curl -LO https://git.federatedsearch.org/frx/frxd/releases/download/v0.1.0/frx-linux-amd64.sha256</pre>
<p class="muted">All releases: <a href="https://git.federatedsearch.org/frx/frxd/releases">git.federatedsearch.org/frx/frxd/releases</a>.
Source and spec (<code>rfc.txt</code>): <a href="https://git.federatedsearch.org/frx/frxd">git.federatedsearch.org/frx/frxd</a>.</p>
</div>
<h2>3. Install</h2>
<div class="card">
<pre class="cmd">sha256sum -c frxd-linux-amd64.sha256
chmod +x frxd-linux-amd64 frx-linux-amd64
sudo mv frxd-linux-amd64 /usr/local/bin/frxd
sudo mv frx-linux-amd64 /usr/local/bin/frx</pre>
<p class="muted">No sudo? Run them in place — each is a single self-contained binary.</p>
</div>
<h2>4. Onboard</h2>
<div class="card">
<pre class="cmd">frxd --onboarding</pre>
<ol>
<li>Paste the credential block from step 1.</li>
<li>The wizard generates your keys, enrolls them with the MA, verifies the signed registry
against the pinned MA key, and wires the federation relays — no domains, DNS, or open ports
needed on your side.</li>
<li>Index a directory and mark what you share:</li>
</ol>
<pre class="cmd">frxd add ~/documents --name docs --shared --exposure metadata
frxd serve</pre>
<p class="muted">Search local-first with <code>frx search "..."</code>; broadcast to the federation with
<code>frx query "..."</code>. Nothing is shared until a collection is explicitly marked
<code>--shared</code>.</p>
</div>
<h2>Who is in</h2>
<p class="muted" id="members">…</p>
<p class="muted"><small>The registry is a signed, versioned, public snapshot: <code>GET /registry.json</code>. Membership is governed by the MA — questions and codes come from there.</small></p>
<script>
const out = document.getElementById("out");
document.getElementById("f").onsubmit = async (e) => {
e.preventDefault();
const label = e.target.label.value, code = e.target.code.value;
const res = await fetch("/v1/signup", {
method: "POST",
headers: {"content-type": "application/json"},
body: JSON.stringify({label, code})
});
const body = await res.json();
out.style.display = "block";
out.textContent = res.ok
? "Membership approved.\n\nNext: download frxd (step 2), install it (step 3), then run `frxd --onboarding` and paste this block:\n\n" + body.credentials + "\n"
: "Failed: " + (body.error || ("http " + res.status));
};
(async () => {
try {
const res = await fetch("/registry.json");
const doc = await res.json();
const ids = doc.members.map((m) => m.id);
document.getElementById("members").textContent = doc.members.length === 0
? "The registry is empty — be the first member."
: doc.members.length + " member(s): " + ids.join(", ") + " · registry v" + doc.version;
} catch (e) {
document.getElementById("members").textContent = "registry unavailable";
}
})();
</script>
</body>
</html>
"##;
pub async fn status(config_path: &Path) -> Result<()> { pub async fn status(config_path: &Path) -> Result<()> {
let config = Config::load(config_path)?; let config = Config::load(config_path)?;
let base = format!("http://{}", config.node.listen); let base = format!("http://{}", config.node.listen);
+35
View File
@@ -13,6 +13,26 @@ pub struct Config {
pub query: QuerySection, pub query: QuerySection,
#[serde(default)] #[serde(default)]
pub index: IndexSection, pub index: IndexSection,
#[serde(default, rename = "match")]
pub matching: MatchSection,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MatchSection {
#[serde(default = "default_min_coverage")]
pub min_coverage: f64,
}
impl Default for MatchSection {
fn default() -> Self {
Self {
min_coverage: default_min_coverage(),
}
}
}
fn default_min_coverage() -> f64 {
0.4
} }
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -27,6 +47,10 @@ pub struct NodeSection {
#[serde(default)] #[serde(default)]
pub ma_key: Option<String>, pub ma_key: Option<String>,
#[serde(default)] #[serde(default)]
pub ca_cert: Option<String>,
#[serde(default)]
pub allow_insecure: bool,
#[serde(default)]
pub dev_bootstrap: bool, pub dev_bootstrap: bool,
#[serde(default = "default_true")] #[serde(default = "default_true")]
pub responder: bool, pub responder: bool,
@@ -132,6 +156,8 @@ impl Config {
relays, relays,
registry: None, registry: None,
ma_key: None, ma_key: None,
ca_cert: None,
allow_insecure: false,
dev_bootstrap: false, dev_bootstrap: false,
responder: true, responder: true,
}, },
@@ -139,6 +165,7 @@ impl Config {
index: IndexSection { index: IndexSection {
data_dir: data_dir.to_string(), data_dir: data_dir.to_string(),
}, },
matching: MatchSection::default(),
} }
} }
@@ -162,6 +189,14 @@ impl Config {
PathBuf::from(&self.index.data_dir) PathBuf::from(&self.index.data_dir)
} }
pub fn insecure_endpoints(&self) -> Vec<String> {
let mut urls = self.node.relays.clone();
if let Some(registry) = &self.node.registry {
urls.push(registry.clone());
}
crate::net::insecure_http_urls(urls)
}
pub fn index_dir(&self) -> PathBuf { pub fn index_dir(&self) -> PathBuf {
self.data_dir().join("index") self.data_dir().join("index")
} }
+67
View File
@@ -0,0 +1,67 @@
use anyhow::Result;
use crate::index::{LocalIndex, SearchHit};
pub struct EngineOutput {
pub hits: Vec<SearchHit>,
pub total: Option<u64>,
}
pub trait SearchEngine: Send + Sync {
fn search(&self, text: &str, budget: usize, only_shared: bool) -> Result<EngineOutput>;
fn doc_count(&self) -> u64;
}
pub struct TantivyEngine {
pub index: LocalIndex,
pub min_coverage: f64,
}
impl SearchEngine for TantivyEngine {
fn search(&self, text: &str, budget: usize, only_shared: bool) -> Result<EngineOutput> {
let (hits, total) =
self.index
.search_with_coverage(text, budget, only_shared, self.min_coverage)?;
Ok(EngineOutput {
hits,
total: Some(total),
})
}
fn doc_count(&self) -> u64 {
self.index.doc_count()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::index::Collection;
use crate::message::EXPOSURE_FULL;
use std::fs;
#[test]
fn tantivy_engine_reports_hits_and_total() {
let temp = tempfile::tempdir().unwrap();
let dir = temp.path().join("corpus");
fs::create_dir_all(&dir).unwrap();
fs::write(dir.join("a.txt"), "rust ownership guide").unwrap();
let engine = TantivyEngine {
index: LocalIndex::open(&temp.path().join("index")).unwrap(),
min_coverage: 0.4,
};
engine
.index
.add_collection(&Collection {
name: "docs".to_string(),
path: dir.display().to_string(),
shared: true,
exposure: EXPOSURE_FULL.to_string(),
})
.unwrap();
let output = engine.search("rust", 5, false).unwrap();
assert_eq!(output.hits.len(), 1);
assert_eq!(output.total, Some(1));
assert_eq!(engine.doc_count(), 1);
}
}
+196 -9
View File
@@ -1,3 +1,4 @@
use std::collections::HashSet;
use std::fs; use std::fs;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Mutex; use std::sync::Mutex;
@@ -6,14 +7,24 @@ use anyhow::{Context, Result};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use tantivy::collector::{Count, TopDocs}; use tantivy::collector::{Count, TopDocs};
use tantivy::directory::MmapDirectory; use tantivy::directory::MmapDirectory;
use tantivy::query::{AllQuery, BooleanQuery, Occur, Query, QueryParser, TermQuery}; use tantivy::query::{
use tantivy::schema::{Field, IndexRecordOption, STORED, STRING, Schema, TEXT, Value}; AllQuery, BooleanQuery, BoostQuery, EmptyQuery, Occur, PhraseQuery, Query, TermQuery,
};
use tantivy::schema::{
Field, IndexRecordOption, STORED, STRING, Schema, TextFieldIndexing, TextOptions, Value,
};
use tantivy::snippet::SnippetGenerator;
use tantivy::tokenizer::TokenStream;
use tantivy::{Index, IndexReader, IndexWriter, TantivyDocument, Term, doc}; use tantivy::{Index, IndexReader, IndexWriter, TantivyDocument, Term, doc};
use crate::extract::{extract_file, summary_of, supported}; use crate::extract::{extract_file, summary_of, supported};
use crate::message::{EXPOSURE_FULL, ResponseItem}; use crate::message::{EXPOSURE_FULL, ResponseItem};
use crate::tokenizer::{TOKENIZER_NAME, analyzer};
const WRITER_BUDGET: usize = 50_000_000; const WRITER_BUDGET: usize = 50_000_000;
const MAX_QUERY_TERMS: usize = 10;
const MAX_COMBOS: usize = 256;
const DEFAULT_MIN_COVERAGE: f64 = 0.4;
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Collection { pub struct Collection {
@@ -102,6 +113,7 @@ impl LocalIndex {
let schema = build_schema(); let schema = build_schema();
let index = Index::open_or_create(MmapDirectory::open(dir)?, schema) let index = Index::open_or_create(MmapDirectory::open(dir)?, schema)
.context("opening tantivy index")?; .context("opening tantivy index")?;
index.tokenizers().register(TOKENIZER_NAME, analyzer());
let schema = index.schema(); let schema = index.schema();
let fields = Fields { let fields = Fields {
url: schema.get_field("url")?, url: schema.get_field("url")?,
@@ -178,17 +190,43 @@ impl LocalIndex {
text: &str, text: &str,
limit: usize, limit: usize,
only_shared: bool, only_shared: bool,
) -> Result<(Vec<SearchHit>, u64)> {
self.search_with_coverage(text, limit, only_shared, DEFAULT_MIN_COVERAGE)
}
pub fn search_with_coverage(
&self,
text: &str,
limit: usize,
only_shared: bool,
min_coverage: f64,
) -> Result<(Vec<SearchHit>, u64)> { ) -> Result<(Vec<SearchHit>, u64)> {
let limit = limit.max(1); let limit = limit.max(1);
self.reader.reload()?; self.reader.reload()?;
let searcher = self.reader.searcher(); let searcher = self.reader.searcher();
let terms = self.query_terms(text);
let user_query: Box<dyn Query> = if text.trim().is_empty() { let user_query: Box<dyn Query> = if text.trim().is_empty() {
Box::new(AllQuery) Box::new(AllQuery)
} else if terms.is_empty() {
Box::new(EmptyQuery)
} else { } else {
let parser = let gate = self.coverage_query(&terms, min_coverage);
QueryParser::for_index(&self.index, vec![self.fields.title, self.fields.body]); let mut parts: Vec<(Occur, Box<dyn Query>)> = vec![(Occur::Must, gate)];
let (query, _errors) = parser.parse_query_lenient(text); if terms.len() >= 2 {
query let phrase_terms: Vec<(usize, Term)> = terms
.iter()
.enumerate()
.map(|(index, term)| (index, Term::from_field_text(self.fields.body, term)))
.collect();
parts.push((
Occur::Should,
Box::new(BoostQuery::new(
Box::new(PhraseQuery::new_with_offset_and_slop(phrase_terms, 1)),
3.0,
)),
));
}
Box::new(BooleanQuery::new(parts))
}; };
let query: Box<dyn Query> = if only_shared { let query: Box<dyn Query> = if only_shared {
let shared_term = TermQuery::new( let shared_term = TermQuery::new(
@@ -204,13 +242,18 @@ impl LocalIndex {
}; };
let total = searcher.search(&*query, &Count)? as u64; let total = searcher.search(&*query, &Count)? as u64;
let top = searcher.search(&*query, &TopDocs::with_limit(limit).order_by_score())?; let top = searcher.search(&*query, &TopDocs::with_limit(limit).order_by_score())?;
let snippet_generator = SnippetGenerator::create(&searcher, &*query, self.fields.body).ok();
let mut hits = Vec::with_capacity(top.len()); let mut hits = Vec::with_capacity(top.len());
for (_score, address) in top { for (_score, address) in top {
let document: TantivyDocument = searcher.doc(address)?; let document: TantivyDocument = searcher.doc(address)?;
let snippet = snippet_generator
.as_ref()
.map(|generator| generator.snippet_from_doc(&document).fragment().to_string())
.filter(|fragment| !fragment.trim().is_empty());
hits.push(SearchHit { hits.push(SearchHit {
url: text_value(&document, self.fields.url), url: text_value(&document, self.fields.url),
title: text_value(&document, self.fields.title), title: text_value(&document, self.fields.title),
summary: text_value(&document, self.fields.summary), summary: snippet.unwrap_or_else(|| text_value(&document, self.fields.summary)),
published: text_value(&document, self.fields.published), published: text_value(&document, self.fields.published),
exposure: text_value(&document, self.fields.exposure), exposure: text_value(&document, self.fields.exposure),
collection: text_value(&document, self.fields.collection), collection: text_value(&document, self.fields.collection),
@@ -221,11 +264,110 @@ impl LocalIndex {
Ok((hits, total)) Ok((hits, total))
} }
fn query_terms(&self, text: &str) -> Vec<String> {
let mut terms = Vec::new();
let mut seen = HashSet::new();
if let Some(mut tokenizer) = self.index.tokenizers().get(TOKENIZER_NAME) {
let mut stream = tokenizer.token_stream(text);
while let Some(token) = stream.next() {
let term = token.text.to_string();
if seen.insert(term.clone()) {
terms.push(term);
}
if terms.len() >= MAX_QUERY_TERMS {
break;
}
}
}
terms
}
fn coverage_query(&self, terms: &[String], min_coverage: f64) -> Box<dyn Query> {
let n = terms.len();
let mut required = if n <= 2 {
n
} else {
((n as f64) * min_coverage).ceil() as usize
};
required = required.clamp(1, n);
let combos = combinations(n, required, MAX_COMBOS);
let mut shoulds: Vec<(Occur, Box<dyn Query>)> = Vec::with_capacity(combos.len());
for combo in combos {
let musts: Vec<(Occur, Box<dyn Query>)> = combo
.iter()
.map(|index| {
let term = terms[*index].as_str();
let term_query: Box<dyn Query> = Box::new(BooleanQuery::new(vec![
(
Occur::Should,
Box::new(BoostQuery::new(
Box::new(TermQuery::new(
Term::from_field_text(self.fields.title, term),
IndexRecordOption::WithFreqs,
)),
2.0,
)),
),
(
Occur::Should,
Box::new(TermQuery::new(
Term::from_field_text(self.fields.body, term),
IndexRecordOption::WithFreqs,
)),
),
]));
(Occur::Must, term_query)
})
.collect();
shoulds.push((Occur::Should, Box::new(BooleanQuery::new(musts))));
}
Box::new(BooleanQuery::new(shoulds))
}
pub fn doc_count(&self) -> u64 { pub fn doc_count(&self) -> u64 {
self.reader.searcher().num_docs() self.reader.searcher().num_docs()
} }
} }
fn binomial(n: usize, k: usize) -> usize {
if k > n {
return 0;
}
let k = k.min(n - k);
let mut result = 1usize;
for index in 0..k {
result = result * (n - index) / (index + 1);
}
result
}
fn combinations(n: usize, start_required: usize, cap: usize) -> Vec<Vec<usize>> {
let mut required = start_required;
while required < n && binomial(n, required) > cap {
required += 1;
}
fn walk(
start: usize,
remaining: usize,
n: usize,
current: &mut Vec<usize>,
out: &mut Vec<Vec<usize>>,
) {
if remaining == 0 {
out.push(current.clone());
return;
}
for index in start..=n - remaining {
current.push(index);
walk(index + 1, remaining - 1, n, current, out);
current.pop();
}
}
let mut out = Vec::new();
walk(0, required, n, &mut Vec::new(), &mut out);
out
}
fn text_value(document: &TantivyDocument, field: Field) -> String { fn text_value(document: &TantivyDocument, field: Field) -> String {
document document
.get_first(field) .get_first(field)
@@ -236,9 +378,16 @@ fn text_value(document: &TantivyDocument, field: Field) -> String {
fn build_schema() -> Schema { fn build_schema() -> Schema {
let mut builder = Schema::builder(); let mut builder = Schema::builder();
let frx_options = TextOptions::default()
.set_indexing_options(
TextFieldIndexing::default()
.set_tokenizer(TOKENIZER_NAME)
.set_index_option(IndexRecordOption::WithFreqsAndPositions),
)
.set_stored();
builder.add_text_field("url", STRING | STORED); builder.add_text_field("url", STRING | STORED);
builder.add_text_field("title", TEXT | STORED); builder.add_text_field("title", frx_options.clone());
builder.add_text_field("body", TEXT | STORED); builder.add_text_field("body", frx_options);
builder.add_text_field("summary", STORED); builder.add_text_field("summary", STORED);
builder.add_text_field("published", STORED); builder.add_text_field("published", STORED);
builder.add_text_field("exposure", STRING | STORED); builder.add_text_field("exposure", STRING | STORED);
@@ -321,6 +470,44 @@ mod tests {
assert!(items[0].content.is_some()); assert!(items[0].content.is_some());
} }
#[test]
fn coverage_gate_rejects_weak_single_term_overlap() {
let temp = tempfile::tempdir().unwrap();
let dir = temp.path().join("corpus");
fs::create_dir_all(&dir).unwrap();
fs::write(dir.join("errors.txt"), "error codes and diagnostics").unwrap();
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
index
.add_collection(&collection("test", &dir, true, EXPOSURE_FULL))
.unwrap();
let (hits, total) = index.search("launch codes", 10, true).unwrap();
assert_eq!(total, 0, "two-term queries require both terms");
assert!(hits.is_empty());
}
#[test]
fn coverage_gate_allows_partial_long_queries() {
let temp = tempfile::tempdir().unwrap();
let dir = temp.path().join("corpus");
fs::create_dir_all(&dir).unwrap();
fs::write(
dir.join("manual.txt"),
"LG DLEX5555 dryer circuit diagram and wiring",
)
.unwrap();
let index = LocalIndex::open(&temp.path().join("index")).unwrap();
index
.add_collection(&collection("test", &dir, true, EXPOSURE_FULL))
.unwrap();
let (hits, total) = index
.search("LG dryer 5555 circuit diagram", 10, true)
.unwrap();
assert_eq!(total, 1, "four of five terms is enough");
assert_eq!(hits.len(), 1);
}
#[test] #[test]
fn reindex_removes_deleted_files() { fn reindex_removes_deleted_files() {
let temp = tempfile::tempdir().unwrap(); let temp = tempfile::tempdir().unwrap();
+4
View File
@@ -1,13 +1,17 @@
pub mod commands; pub mod commands;
pub mod config; pub mod config;
pub mod crypto; pub mod crypto;
pub mod engine;
pub mod extract; pub mod extract;
pub mod index; pub mod index;
pub mod message; pub mod message;
pub mod net;
pub mod node; pub mod node;
pub mod onboard;
pub mod registry; pub mod registry;
pub mod relay; pub mod relay;
pub mod render; pub mod render;
pub mod tokenizer;
pub const VERSION: &str = env!("CARGO_PKG_VERSION"); pub const VERSION: &str = env!("CARGO_PKG_VERSION");
pub const PROTOCOL: &str = "FRX/0.5"; pub const PROTOCOL: &str = "FRX/0.5";
+47 -7
View File
@@ -5,19 +5,23 @@ use clap::{Parser, Subcommand, ValueEnum};
use frxd::config::Config; use frxd::config::Config;
use frxd::crypto::Keypair; use frxd::crypto::Keypair;
use frxd::message::{EXPOSURE_FULL, EXPOSURE_METADATA}; use frxd::message::{EXPOSURE_FULL, EXPOSURE_METADATA};
use frxd::{commands, node, relay}; use frxd::{commands, node, onboard, relay};
#[derive(Parser)] #[derive(Parser)]
#[command( #[command(
name = "frxd", name = "frxd",
version, version,
about = "FRX member node — querier, responder, and local index (Draft 0.4)" about = "FRX member node — querier, responder, and local index (Draft 0.5)"
)] )]
struct Cli { struct Cli {
#[arg(long, global = true, default_value = "frxd.toml")] #[arg(long, global = true, default_value = "frxd.toml")]
config: PathBuf, config: PathBuf,
#[arg(long)]
onboarding: bool,
#[arg(long)]
listen: Option<String>,
#[command(subcommand)] #[command(subcommand)]
command: Command, command: Option<Command>,
} }
#[derive(Subcommand)] #[derive(Subcommand)]
@@ -39,6 +43,10 @@ enum Command {
registry: Option<String>, registry: Option<String>,
#[arg(long)] #[arg(long)]
ma_key: Option<String>, ma_key: Option<String>,
#[arg(long)]
ca_cert: Option<String>,
#[arg(long)]
allow_insecure: bool,
}, },
Add { Add {
path: PathBuf, path: PathBuf,
@@ -78,6 +86,10 @@ enum Command {
registry: Option<String>, registry: Option<String>,
#[arg(long)] #[arg(long)]
ma_key: Option<String>, ma_key: Option<String>,
#[arg(long)]
ca_cert: Option<String>,
#[arg(long)]
allow_insecure: bool,
#[arg(long, default_value_t = 3)] #[arg(long, default_value_t = 3)]
max_hops: usize, max_hops: usize,
}, },
@@ -131,7 +143,10 @@ enum KeyCommand {
#[derive(Subcommand)] #[derive(Subcommand)]
enum RegistryCommand { enum RegistryCommand {
Init, Init {
#[arg(long, default_value = "frx.invalid")]
zone: String,
},
Add { Add {
id: String, id: String,
pubkey: String, pubkey: String,
@@ -172,6 +187,10 @@ enum RegistryCommand {
Serve { Serve {
#[arg(long, default_value = "127.0.0.1:7800")] #[arg(long, default_value = "127.0.0.1:7800")]
listen: String, listen: String,
#[arg(long)]
signup_code: Option<String>,
#[arg(long)]
registry_url: Option<String>,
}, },
} }
@@ -184,7 +203,16 @@ enum Exposure {
#[tokio::main] #[tokio::main]
async fn main() -> Result<()> { async fn main() -> Result<()> {
let cli = Cli::parse(); let cli = Cli::parse();
match cli.command { if cli.onboarding {
let mut input = std::io::stdin().lock();
let mut output = std::io::stdout().lock();
onboard::run(&mut input, &mut output, &cli.config, cli.listen.as_deref()).await?;
return Ok(());
}
let Some(command) = cli.command else {
bail!("no subcommand given (try --onboarding or --help)");
};
match command {
Command::Init { Command::Init {
name, name,
listen, listen,
@@ -194,6 +222,8 @@ async fn main() -> Result<()> {
id, id,
registry, registry,
ma_key, ma_key,
ca_cert,
allow_insecure,
} => { } => {
if cli.config.exists() && !force { if cli.config.exists() && !force {
bail!( bail!(
@@ -208,6 +238,8 @@ async fn main() -> Result<()> {
config.node.id = id; config.node.id = id;
config.node.registry = registry; config.node.registry = registry;
config.node.ma_key = ma_key; config.node.ma_key = ma_key;
config.node.ca_cert = ca_cert;
config.node.allow_insecure = allow_insecure;
if config.node.registry.is_none() { if config.node.registry.is_none() {
config.node.dev_bootstrap = true; config.node.dev_bootstrap = true;
println!( println!(
@@ -262,6 +294,8 @@ async fn main() -> Result<()> {
url, url,
registry, registry,
ma_key, ma_key,
ca_cert,
allow_insecure,
max_hops, max_hops,
} => { } => {
let options = relay::RelayOptions { let options = relay::RelayOptions {
@@ -270,6 +304,8 @@ async fn main() -> Result<()> {
url, url,
registry, registry,
ma_key, ma_key,
ca_cert,
allow_insecure,
max_hops, max_hops,
}; };
let (listener, addr) = relay::bind(&listen).await?; let (listener, addr) = relay::bind(&listen).await?;
@@ -293,7 +329,7 @@ async fn main() -> Result<()> {
KeyCommand::Rotate => commands::key_rotate(&cli.config)?, KeyCommand::Rotate => commands::key_rotate(&cli.config)?,
}, },
Command::Registry { dir, command } => match command { Command::Registry { dir, command } => match command {
RegistryCommand::Init => commands::registry_init(&dir)?, RegistryCommand::Init { zone } => commands::registry_init(&dir, &zone)?,
RegistryCommand::Add { RegistryCommand::Add {
id, id,
pubkey, pubkey,
@@ -320,7 +356,11 @@ async fn main() -> Result<()> {
RegistryCommand::List => commands::registry_list(&dir)?, RegistryCommand::List => commands::registry_list(&dir)?,
RegistryCommand::SetRelays { relays } => commands::registry_set_relays(&dir, &relays)?, RegistryCommand::SetRelays { relays } => commands::registry_set_relays(&dir, &relays)?,
RegistryCommand::Show => commands::registry_show(&dir)?, RegistryCommand::Show => commands::registry_show(&dir)?,
RegistryCommand::Serve { listen } => commands::registry_serve(&dir, &listen).await?, RegistryCommand::Serve {
listen,
signup_code,
registry_url,
} => commands::registry_serve(&dir, &listen, signup_code, registry_url).await?,
}, },
Command::Aggregates { Command::Aggregates {
from, from,
+13 -7
View File
@@ -102,17 +102,23 @@ pub struct ResponseBody {
pub fn build_response( pub fn build_response(
qid: &str, qid: &str,
hits: Vec<ResponseItem>, hits: Vec<ResponseItem>,
total: u64, total: Option<u64>,
max_results: usize, max_results: usize,
) -> ResponseBody { ) -> ResponseBody {
let max_results = max_results.max(1); let max_results = max_results.max(1);
let mut results = hits; let mut results = hits;
results.truncate(max_results); results.truncate(max_results);
let more_available = total.saturating_sub(results.len() as u64); let (truncated, more_available) = match total {
Some(total) => {
let more = total.saturating_sub(results.len() as u64);
(more > 0, more)
}
None => (true, 0),
};
ResponseBody { ResponseBody {
qid: qid.to_string(), qid: qid.to_string(),
results, results,
truncated: more_available > 0, truncated,
more_available, more_available,
cursor: None, cursor: None,
} }
@@ -153,24 +159,24 @@ mod tests {
#[test] #[test]
fn max_results_is_respected() { fn max_results_is_respected() {
let response = build_response("q1", vec![item(1), item(2), item(3)], 10, 2); let response = build_response("q1", vec![item(1), item(2), item(3)], Some(10), 2);
assert_eq!(response.results.len(), 2); assert_eq!(response.results.len(), 2);
} }
#[test] #[test]
fn truncation_is_honest() { fn truncation_is_honest() {
let response = build_response("q1", vec![item(1), item(2), item(3)], 10, 2); let response = build_response("q1", vec![item(1), item(2), item(3)], Some(10), 2);
assert!(response.truncated); assert!(response.truncated);
assert_eq!(response.more_available, 8); assert_eq!(response.more_available, 8);
let response = build_response("q2", vec![item(1), item(2)], 2, 5); let response = build_response("q2", vec![item(1), item(2)], Some(2), 5);
assert!(!response.truncated); assert!(!response.truncated);
assert_eq!(response.more_available, 0); assert_eq!(response.more_available, 0);
} }
#[test] #[test]
fn response_carries_no_scores() { fn response_carries_no_scores() {
let response = build_response("q1", vec![item(1)], 1, 5); let response = build_response("q1", vec![item(1)], Some(1), 5);
let json = serde_json::to_string(&response).unwrap(); let json = serde_json::to_string(&response).unwrap();
assert!(!json.contains("score")); assert!(!json.contains("score"));
assert!(!json.contains("relevance")); assert!(!json.contains("relevance"));
+79
View File
@@ -0,0 +1,79 @@
use std::path::Path;
use std::time::Duration;
use anyhow::{Context, Result};
use reqwest::Client;
pub fn build_client(ca_cert: Option<&Path>, timeout: Duration) -> Result<Client> {
let mut builder = Client::builder().timeout(timeout);
if let Some(path) = ca_cert {
let pem = std::fs::read(path)
.with_context(|| format!("reading CA certificate {}", path.display()))?;
let certificates =
reqwest::Certificate::from_pem_bundle(&pem).context("parsing CA certificate bundle")?;
for certificate in certificates {
builder = builder.add_root_certificate(certificate);
}
}
builder.build().context("building http client")
}
pub fn is_loopback_url(url: &str) -> bool {
let rest = url.split_once("://").map(|(_, rest)| rest).unwrap_or(url);
let host_port = rest.split(['/', '?', '#']).next().unwrap_or("");
let host = if let Some(stripped) = host_port.strip_prefix('[') {
stripped.split(']').next().unwrap_or("")
} else {
host_port.split(':').next().unwrap_or("")
};
matches!(host, "127.0.0.1" | "localhost" | "::1")
}
pub fn insecure_http_urls<I, S>(urls: I) -> Vec<String>
where
I: IntoIterator<Item = S>,
S: AsRef<str>,
{
urls.into_iter()
.map(|url| url.as_ref().to_string())
.filter(|url| url.starts_with("http://") && !is_loopback_url(url))
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn loopback_detection() {
assert!(is_loopback_url("http://127.0.0.1:7700"));
assert!(is_loopback_url("http://localhost:7700/v1"));
assert!(is_loopback_url("https://[::1]:7700"));
assert!(!is_loopback_url("http://10.0.0.1:7700"));
assert!(!is_loopback_url("https://relay.example.com"));
assert!(!is_loopback_url("http://relay.example.com"));
}
#[test]
fn insecure_filter_keeps_only_plain_non_loopback() {
let urls = vec![
"http://127.0.0.1:7700",
"https://relay.example.com",
"http://relay.example.com",
"http://10.0.0.1:7700",
];
assert_eq!(
insecure_http_urls(urls),
vec!["http://relay.example.com", "http://10.0.0.1:7700"]
);
}
#[test]
fn missing_ca_file_is_an_error() {
let result = build_client(
Some(Path::new("/nonexistent/ca.pem")),
Duration::from_secs(1),
);
assert!(result.is_err());
}
}
+29 -13
View File
@@ -18,6 +18,7 @@ use tokio::task::JoinHandle;
use crate::config::{CLASS_ENRICHMENT, Config, Member, load_members}; use crate::config::{CLASS_ENRICHMENT, Config, Member, load_members};
use crate::crypto::{Keypair, now_ts, poll_signing_bytes}; use crate::crypto::{Keypair, now_ts, poll_signing_bytes};
use crate::engine::{SearchEngine, TantivyEngine};
use crate::index::{LocalIndex, SearchHit, response_items}; use crate::index::{LocalIndex, SearchHit, response_items};
use crate::message::{ use crate::message::{
AggregateBody, Envelope, QueryBody, ResponseBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY, AggregateBody, Envelope, QueryBody, ResponseBody, ResponseItem, TYPE_AGGREGATE, TYPE_QUERY,
@@ -34,7 +35,7 @@ struct Aggregates {
pub struct Node { pub struct Node {
pub config: Config, pub config: Config,
pub key: Keypair, pub key: Keypair,
index: LocalIndex, engine: Arc<dyn SearchEngine>,
members: RwLock<Vec<Member>>, members: RwLock<Vec<Member>>,
members_mtime: Mutex<Option<SystemTime>>, members_mtime: Mutex<Option<SystemTime>>,
registry: Option<Arc<Watcher>>, registry: Option<Arc<Watcher>>,
@@ -113,16 +114,28 @@ pub struct LocalPart {
impl Node { impl Node {
pub fn open(config: Config) -> Result<Arc<Self>> { pub fn open(config: Config) -> Result<Arc<Self>> {
let key = config.load_key()?; let key = config.load_key()?;
let index = LocalIndex::open(&config.index_dir())?; let engine: Arc<dyn SearchEngine> = Arc::new(TantivyEngine {
index: LocalIndex::open(&config.index_dir())?,
min_coverage: config.matching.min_coverage,
});
let members_path = config.members_path(); let members_path = config.members_path();
let members = load_members(&members_path)?; let members = load_members(&members_path)?;
let members_mtime = fs::metadata(&members_path) let members_mtime = fs::metadata(&members_path)
.and_then(|metadata| metadata.modified()) .and_then(|metadata| metadata.modified())
.ok(); .ok();
let client = reqwest::Client::builder() if !config.node.allow_insecure {
.timeout(Duration::from_secs(15)) let insecure = config.insecure_endpoints();
.build() if !insecure.is_empty() {
.context("building http client")?; return Err(anyhow!(
"refusing plain http endpoints (use https, configure ca_cert, or set allow_insecure for private networks): {}",
insecure.join(", ")
));
}
}
let client = crate::net::build_client(
config.node.ca_cert.as_deref().map(std::path::Path::new),
Duration::from_secs(15),
)?;
let registry = match ( let registry = match (
config.node.registry.as_deref(), config.node.registry.as_deref(),
config.node.ma_key.as_deref(), config.node.ma_key.as_deref(),
@@ -146,10 +159,10 @@ impl Node {
Ok(Arc::new(Self { Ok(Arc::new(Self {
config, config,
key, key,
index,
members: RwLock::new(members), members: RwLock::new(members),
members_mtime: Mutex::new(members_mtime), members_mtime: Mutex::new(members_mtime),
registry, registry,
engine,
enc_secret, enc_secret,
enc_public, enc_public,
aggregates: Mutex::new(Aggregates::default()), aggregates: Mutex::new(Aggregates::default()),
@@ -220,7 +233,7 @@ impl Node {
} }
pub fn doc_count(&self) -> u64 { pub fn doc_count(&self) -> u64 {
self.index.doc_count() self.engine.doc_count()
} }
pub fn identifier(&self) -> String { pub fn identifier(&self) -> String {
@@ -240,7 +253,8 @@ impl Node {
} }
pub fn local_search(&self, text: &str, limit: usize) -> Result<(Vec<SearchHit>, u64)> { pub fn local_search(&self, text: &str, limit: usize) -> Result<(Vec<SearchHit>, u64)> {
self.index.search(text, limit, false) let output = self.engine.search(text, limit, false)?;
Ok((output.hits, output.total.unwrap_or(0)))
} }
pub async fn start(config: Config) -> Result<NodeHandle> { pub async fn start(config: Config) -> Result<NodeHandle> {
@@ -301,7 +315,9 @@ impl Node {
let max = max_results.unwrap_or(self.config.query.max_results).max(1); let max = max_results.unwrap_or(self.config.query.max_results).max(1);
let query = QueryBody::new(text, max); let query = QueryBody::new(text, max);
let qid = query.qid.clone(); let qid = query.qid.clone();
let (local_hits, local_total) = self.index.search(text, max, false)?; let output = self.engine.search(text, max, false)?;
let local_hits = output.hits;
let local_total = output.total.unwrap_or(0);
let local_items = response_items(&local_hits); let local_items = response_items(&local_hits);
let mut responses = Vec::new(); let mut responses = Vec::new();
if network { if network {
@@ -551,11 +567,11 @@ impl Node {
async fn respond(&self, query: &QueryBody, querier: &str, relay: &str) -> Result<()> { async fn respond(&self, query: &QueryBody, querier: &str, relay: &str) -> Result<()> {
let max = query.budget.max_results.clamp(1, 1000); let max = query.budget.max_results.clamp(1, 1000);
let (hits, total) = self.index.search(&query.text, max, true)?; let output = self.engine.search(&query.text, max, true)?;
if hits.is_empty() { if output.hits.is_empty() {
return Ok(()); return Ok(());
} }
let body = build_response(&query.qid, response_items(&hits), total, max); let body = build_response(&query.qid, response_items(&output.hits), output.total, max);
self.send_payload(querier, TYPE_RESPONSE, serde_json::to_value(&body)?, relay) self.send_payload(querier, TYPE_RESPONSE, serde_json::to_value(&body)?, relay)
.await .await
} }
+284
View File
@@ -0,0 +1,284 @@
use std::io::{BufRead, Write};
use std::path::{Path, PathBuf};
use std::time::Duration;
use anyhow::{Context, Result, anyhow, bail};
use serde_json::Value;
use crate::config::Config;
use crate::crypto::{Keypair, generate_enc_keypair, now_ts};
use crate::registry::{SignedRegistry, authorized_keys, load_registry, verify_registry};
pub async fn run(
reader: &mut impl BufRead,
writer: &mut impl Write,
config_path: &Path,
listen_override: Option<&str>,
) -> Result<()> {
writeln!(writer, "FRX onboarding")?;
writeln!(
writer,
"You need credentials from your FRX membership authority (issued by the membership site or your operator)."
)?;
let path_answer = prompt(
writer,
reader,
"config file",
&config_path.display().to_string(),
)?;
let config_path = PathBuf::from(path_answer);
if config_path.exists() {
bail!(
"config {} already exists; choose another path or remove it",
config_path.display()
);
}
let block = prompt(
writer,
reader,
"paste credential block (id=.. token=.. registry=.. ma_key=..), or press enter to enter fields",
"",
)?;
let Credentials {
id,
token,
registry,
ma_key,
} = if block.trim().is_empty() {
Credentials {
id: prompt(writer, reader, "member id (e.g. alice.frx.example)", "")?,
token: prompt(
writer,
reader,
"invite token (leave empty if the MA added your key out of band)",
"",
)?,
registry: prompt(writer, reader, "registry URL or file path", "")?,
ma_key: prompt(writer, reader, "MA key (pubkey hex)", "")?,
}
} else {
parse_block(&block)?
};
if id.is_empty() || registry.is_empty() || ma_key.is_empty() {
bail!("member id, registry, and MA key are required");
}
let (relays_from_registry, authorized) = {
let signed = if registry.starts_with("http://") || registry.starts_with("https://") {
let client = crate::net::build_client(None, Duration::from_secs(10))?;
fetch_registry(&client, &registry).await?
} else {
load_registry(Path::new(&registry))?
};
verify_registry(&signed, &ma_key)?;
(
signed.doc.relays.clone(),
authorized_keys(&signed, now_ts()),
)
};
let key = Keypair::generate();
let (enc_secret, enc_public) = generate_enc_keypair();
if registry.starts_with("http://") || registry.starts_with("https://") {
if token.is_empty() {
if !authorized.contains_key(&key.public_hex()) {
writeln!(
writer,
"note: your key {} is not yet authorized; the MA must run: frxd registry add {id} {} --enc-key {enc_public}",
key.public_hex(),
key.public_hex()
)?;
}
} else {
let client = crate::net::build_client(None, Duration::from_secs(10))?;
let enroll_url = format!("{}/v1/enroll", registry_base(&registry));
let response = client
.post(&enroll_url)
.json(&serde_json::json!({
"id": id,
"token": token,
"pubkey": key.public_hex(),
"enc_key": enc_public,
}))
.send()
.await
.context("calling enrollment endpoint")?;
let status = response.status();
let body: Value = response.json().await.unwrap_or_default();
if !status.is_success() {
let message = body
.get("error")
.and_then(Value::as_str)
.unwrap_or("unknown error");
bail!("enrollment failed: {message}");
}
let signed = fetch_registry(&client, &registry).await?;
verify_registry(&signed, &ma_key)?;
if !authorized_keys(&signed, now_ts()).contains_key(&key.public_hex()) {
bail!("enrollment accepted but the registry does not yet authorize the key");
}
writeln!(writer, "enrolled {id} with the MA")?;
}
} else if !authorized.contains_key(&key.public_hex()) {
writeln!(
writer,
"your pubkey is {}; the MA must run: frxd registry add {id} {} --enc-key {enc_public}",
key.public_hex(),
key.public_hex()
)?;
}
let relay_default = if relays_from_registry.is_empty() {
"http://127.0.0.1:7700".to_string()
} else {
relays_from_registry.join(",")
};
let relays_answer = prompt(
writer,
reader,
"relay URLs (comma separated)",
&relay_default,
)?;
let relays: Vec<String> = relays_answer
.split(',')
.map(|relay| relay.trim().to_string())
.filter(|relay| !relay.is_empty())
.collect();
if relays.is_empty() {
bail!("at least one relay is required");
}
let data_dir = prompt(writer, reader, "data directory", "./frx-data")?;
let listen = listen_override
.map(str::to_string)
.unwrap_or_else(|| pick_listen("127.0.0.1:7701"));
let mut config = Config::new(&id, &listen, relays.clone(), &data_dir);
config.node.id = Some(id.clone());
config.node.registry = Some(registry.clone());
config.node.ma_key = Some(ma_key.clone());
config.save_key(&key)?;
config.save_enc_key(&enc_secret)?;
config.save(&config_path)?;
let share_dir = prompt(
writer,
reader,
"directory to share (empty to skip; you can add collections later with `frxd add`)",
"",
)?;
if !share_dir.trim().is_empty() {
let exposure = prompt(writer, reader, "exposure for the collection", "metadata")?;
crate::commands::add(
&config_path,
Path::new(share_dir.trim()),
None,
true,
&exposure,
)?;
}
writeln!(writer)?;
writeln!(writer, "Onboarded as {id}")?;
writeln!(writer, " config: {}", config_path.display())?;
writeln!(writer, " registry: {registry}")?;
writeln!(writer, " relays: {}", relays.join(", "))?;
writeln!(writer, " listening: {listen}")?;
writeln!(
writer,
"Next: frxd serve, then frx query \"...\" to broadcast."
)?;
let start = prompt(writer, reader, "start serving now?", "n")?;
if start == "y" || start.eq_ignore_ascii_case("yes") {
let config = Config::load(&config_path)?;
let handle = crate::node::Node::start(config).await?;
writeln!(writer, "frxd listening on http://{}", handle.addr)?;
writeln!(writer, "ctrl-c to stop")?;
tokio::signal::ctrl_c().await?;
}
Ok(())
}
struct Credentials {
id: String,
token: String,
registry: String,
ma_key: String,
}
fn parse_block(block: &str) -> Result<Credentials> {
let mut id = None;
let mut token = None;
let mut registry = None;
let mut ma_key = None;
for part in block.split_whitespace() {
let Some((key, value)) = part.split_once('=') else {
continue;
};
match key {
"id" => id = Some(value.to_string()),
"token" => token = Some(value.to_string()),
"registry" => registry = Some(value.to_string()),
"ma_key" => ma_key = Some(value.to_string()),
_ => {}
}
}
Ok(Credentials {
id: id.ok_or_else(|| anyhow!("credential block is missing id"))?,
token: token.unwrap_or_default(),
registry: registry.ok_or_else(|| anyhow!("credential block is missing registry"))?,
ma_key: ma_key.ok_or_else(|| anyhow!("credential block is missing ma_key"))?,
})
}
fn pick_listen(preferred: &str) -> String {
if std::net::TcpListener::bind(preferred).is_ok() {
return preferred.to_string();
}
let fallback = std::net::TcpListener::bind("127.0.0.1:0").expect("bind ephemeral port");
fallback.local_addr().expect("local addr").to_string()
}
fn registry_base(registry_url: &str) -> String {
match registry_url.rfind('/') {
Some(index) => registry_url[..index].to_string(),
None => registry_url.to_string(),
}
}
async fn fetch_registry(client: &reqwest::Client, url: &str) -> Result<SignedRegistry> {
let response = client
.get(url)
.send()
.await
.with_context(|| format!("fetching registry {url}"))?;
if !response.status().is_success() {
bail!("registry fetch failed: {}", response.status());
}
let signed: SignedRegistry = response.json().await.context("parsing registry")?;
Ok(signed)
}
fn prompt(
writer: &mut impl Write,
reader: &mut impl BufRead,
label: &str,
default: &str,
) -> Result<String> {
if default.is_empty() {
write!(writer, "{label}: ")?;
} else {
write!(writer, "{label} [{default}]: ")?;
}
writer.flush()?;
let mut line = String::new();
reader.read_line(&mut line)?;
let line = line.trim();
Ok(if line.is_empty() {
default.to_string()
} else {
line.to_string()
})
}
+68
View File
@@ -35,12 +35,79 @@ pub struct RegistryDoc {
pub version: u64, pub version: u64,
pub issued_at: u64, pub issued_at: u64,
pub ma_key: String, pub ma_key: String,
#[serde(default = "default_zone")]
pub zone: String,
#[serde(default)] #[serde(default)]
pub members: Vec<RegistryMember>, pub members: Vec<RegistryMember>,
#[serde(default)] #[serde(default)]
pub relays: Vec<String>, pub relays: Vec<String>,
} }
fn default_zone() -> String {
"frx.invalid".to_string()
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Invite {
pub id: String,
pub token: String,
#[serde(default)]
pub used: bool,
#[serde(default)]
pub expires: u64,
}
pub fn invites_path(dir: &Path) -> PathBuf {
dir.join("invites.json")
}
pub fn load_invites(path: &Path) -> Result<Vec<Invite>> {
if !path.exists() {
return Ok(Vec::new());
}
let raw = fs::read_to_string(path).context("reading invites")?;
serde_json::from_str(&raw).context("parsing invites")
}
pub fn save_invites(path: &Path, invites: &[Invite]) -> Result<()> {
fs::write(path, serde_json::to_string_pretty(invites)?)?;
Ok(())
}
pub fn create_invite(dir: &Path, id: &str, ttl_secs: u64) -> Result<Invite> {
let path = invites_path(dir);
let mut invites = load_invites(&path)?;
let invite = Invite {
id: id.to_string(),
token: crate::crypto::random_nonce(),
used: false,
expires: now_ts() + ttl_secs,
};
invites.push(invite.clone());
save_invites(&path, &invites)?;
Ok(invite)
}
pub fn redeem_invite(dir: &Path, id: &str, token: &str) -> Result<()> {
let path = invites_path(dir);
let mut invites = load_invites(&path)?;
let Some(invite) = invites
.iter_mut()
.find(|invite| invite.id == id && invite.token == token)
else {
return Err(anyhow!("unknown invite"));
};
if invite.used {
return Err(anyhow!("invite already redeemed"));
}
if invite.expires < now_ts() {
return Err(anyhow!("invite expired"));
}
invite.used = true;
save_invites(&path, &invites)?;
Ok(())
}
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SignedRegistry { pub struct SignedRegistry {
#[serde(flatten)] #[serde(flatten)]
@@ -254,6 +321,7 @@ mod tests {
version: 1, version: 1,
issued_at: now_ts(), issued_at: now_ts(),
ma_key: String::new(), ma_key: String::new(),
zone: "frx.invalid".to_string(),
members, members,
relays: Vec::new(), relays: Vec::new(),
}, },
+55 -3
View File
@@ -31,6 +31,8 @@ pub struct RelayOptions {
pub url: Option<String>, pub url: Option<String>,
pub registry: Option<String>, pub registry: Option<String>,
pub ma_key: Option<String>, pub ma_key: Option<String>,
pub ca_cert: Option<String>,
pub allow_insecure: bool,
pub max_hops: usize, pub max_hops: usize,
} }
@@ -42,6 +44,8 @@ impl Default for RelayOptions {
url: None, url: None,
registry: None, registry: None,
ma_key: None, ma_key: None,
ca_cert: None,
allow_insecure: false,
max_hops: 3, max_hops: 3,
} }
} }
@@ -84,6 +88,23 @@ impl Relay {
if !options.peers.is_empty() && options.url.is_none() { if !options.peers.is_empty() && options.url.is_none() {
return Err(anyhow!("--url is required when --peer is set")); return Err(anyhow!("--url is required when --peer is set"));
} }
if !options.allow_insecure {
let mut urls = options.peers.clone();
if let Some(registry) = &options.registry {
urls.push(registry.clone());
}
let insecure = crate::net::insecure_http_urls(urls);
if !insecure.is_empty() {
return Err(anyhow!(
"refusing plain http endpoints (use https, configure --ca-cert, or set --allow-insecure for private networks): {}",
insecure.join(", ")
));
}
}
let client = crate::net::build_client(
options.ca_cert.as_deref().map(std::path::Path::new),
Duration::from_secs(5),
)?;
let relay = Arc::new(Self { let relay = Arc::new(Self {
inner: Mutex::new(Inner { inner: Mutex::new(Inner {
members: HashMap::new(), members: HashMap::new(),
@@ -93,9 +114,7 @@ impl Relay {
}), }),
options, options,
registry, registry,
client: reqwest::Client::builder() client,
.timeout(Duration::from_secs(5))
.build()?,
notify: Notify::new(), notify: Notify::new(),
}); });
if let Some(watcher) = &relay.registry { if let Some(watcher) = &relay.registry {
@@ -520,3 +539,36 @@ fn backpressure(status: StatusCode, missed: u64) -> Response {
) )
.into_response() .into_response()
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn refuses_plain_http_peers_off_loopback() {
let options = RelayOptions {
peers: vec!["http://10.0.0.1:7700".to_string()],
url: Some("http://10.0.0.1:7700".to_string()),
..Default::default()
};
assert!(Relay::new(options).is_err());
let options = RelayOptions {
peers: vec!["http://10.0.0.1:7700".to_string()],
url: Some("http://10.0.0.1:7700".to_string()),
allow_insecure: true,
..Default::default()
};
assert!(Relay::new(options).is_ok());
}
#[test]
fn loopback_peers_need_no_opt_in() {
let options = RelayOptions {
peers: vec!["http://127.0.0.1:7701".to_string()],
url: Some("http://127.0.0.1:7700".to_string()),
..Default::default()
};
assert!(Relay::new(options).is_ok());
}
}
+131
View File
@@ -0,0 +1,131 @@
use tantivy::tokenizer::{
AsciiFoldingFilter, Language, LowerCaser, RemoveLongFilter, Stemmer, StopWordFilter,
TextAnalyzer, Token, TokenStream, Tokenizer,
};
#[derive(Clone, Default)]
pub struct BoundaryTokenizer {
token: Token,
}
impl BoundaryTokenizer {
pub fn new() -> Self {
Self {
token: Token::default(),
}
}
}
impl Tokenizer for BoundaryTokenizer {
type TokenStream<'a> = BoundaryTokenStream<'a>;
fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {
self.token.reset();
BoundaryTokenStream {
text,
chars: text.char_indices().peekable(),
token: &mut self.token,
position: 0,
}
}
}
pub struct BoundaryTokenStream<'a> {
text: &'a str,
chars: std::iter::Peekable<std::str::CharIndices<'a>>,
token: &'a mut Token,
position: usize,
}
impl TokenStream for BoundaryTokenStream<'_> {
fn advance(&mut self) -> bool {
let mut start = None;
let mut alphabetic = None;
while let Some(&(index, ch)) = self.chars.peek() {
if !ch.is_alphanumeric() {
if start.is_none() {
self.chars.next();
continue;
}
break;
}
match alphabetic {
None => {
start = Some(index);
alphabetic = Some(ch.is_alphabetic());
self.chars.next();
}
Some(prev) if prev == ch.is_alphabetic() => {
self.chars.next();
}
Some(_) => break,
}
}
let Some(start) = start else {
return false;
};
let end = self
.chars
.peek()
.map(|&(index, _)| index)
.unwrap_or(self.text.len());
self.token.text = self.text[start..end].to_string();
self.token.offset_from = start;
self.token.offset_to = end;
self.token.position = self.position;
self.position += 1;
true
}
fn token(&self) -> &Token {
self.token
}
fn token_mut(&mut self) -> &mut Token {
self.token
}
}
pub fn analyzer() -> TextAnalyzer {
TextAnalyzer::builder(BoundaryTokenizer::new())
.filter(LowerCaser)
.filter(AsciiFoldingFilter)
.filter(StopWordFilter::new(Language::English).expect("english stopwords"))
.filter(Stemmer::new(Language::English))
.filter(RemoveLongFilter::limit(255))
.build()
}
pub const TOKENIZER_NAME: &str = "frx";
#[cfg(test)]
mod tests {
use super::*;
fn tokens(text: &str) -> Vec<String> {
let mut analyzer = analyzer();
let mut stream = analyzer.token_stream(text);
let mut out = Vec::new();
while let Some(token) = stream.next() {
out.push(token.text.clone());
}
out
}
#[test]
fn splits_model_numbers_and_folds_case() {
let tokens = tokens("DLEX5555 Café");
assert!(tokens.contains(&"dlex".to_string()), "{tokens:?}");
assert!(tokens.contains(&"5555".to_string()), "{tokens:?}");
assert!(tokens.contains(&"cafe".to_string()), "{tokens:?}");
}
#[test]
fn stems_and_drops_stopwords() {
let tokens = tokens("the dryers are searching");
assert!(!tokens.contains(&"the".to_string()), "{tokens:?}");
assert!(!tokens.contains(&"are".to_string()), "{tokens:?}");
assert!(tokens.iter().any(|t| t.starts_with("dryer")), "{tokens:?}");
assert!(tokens.iter().any(|t| t.starts_with("search")), "{tokens:?}");
}
}
+34
View File
@@ -324,6 +324,40 @@ fn cli_key_rotation() {
assert!(data.join("key.hex.bak").exists()); assert!(data.join("key.hex.bak").exists());
} }
#[test]
fn cli_relay_refuses_plain_http_off_loopback() {
let output = frxd()
.args([
"relay",
"--listen",
"127.0.0.1:0",
"--url",
"http://10.0.0.1:1",
"--peer",
"http://10.0.0.1:2",
])
.output()
.unwrap();
assert!(!output.status.success());
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(stderr.contains("allow-insecure"), "{stderr}");
let port = common::free_port();
let _service = spawn_service(
&[
"relay".to_string(),
"--listen".to_string(),
format!("127.0.0.1:{port}"),
"--url".to_string(),
"http://10.0.0.1:1".to_string(),
"--peer".to_string(),
"http://10.0.0.1:2".to_string(),
"--allow-insecure".to_string(),
],
"relay listening",
);
}
#[test] #[test]
fn cli_full_network_pipeline() { fn cli_full_network_pipeline() {
let root = tempfile::tempdir().unwrap(); let root = tempfile::tempdir().unwrap();
+3
View File
@@ -18,6 +18,8 @@ pub fn config_for(dir: &Path, name: &str, relay_url: &str) -> Config {
relays: vec![relay_url.to_string()], relays: vec![relay_url.to_string()],
registry: None, registry: None,
ma_key: None, ma_key: None,
ca_cert: None,
allow_insecure: false,
dev_bootstrap: true, dev_bootstrap: true,
responder: true, responder: true,
}, },
@@ -28,6 +30,7 @@ pub fn config_for(dir: &Path, name: &str, relay_url: &str) -> Config {
index: IndexSection { index: IndexSection {
data_dir: dir.join("data").display().to_string(), data_dir: dir.join("data").display().to_string(),
}, },
matching: Default::default(),
}; };
config.save_key(&Keypair::generate()).unwrap(); config.save_key(&Keypair::generate()).unwrap();
config config
+27
View File
@@ -448,6 +448,33 @@ async fn rotated_keys_are_accepted_through_previous_listing() {
assert!(revoked.is_empty(), "revoked key was still accepted"); assert!(revoked.is_empty(), "revoked key was still accepted");
} }
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn plain_http_transport_is_refused_off_loopback() {
let root = tempfile::tempdir().unwrap();
let mut config = config_for(&root.path().join("alice"), "alice", "http://10.0.0.1:1");
let refused = Node::start(config.clone()).await;
assert!(
refused.is_err(),
"plain http to a non-loopback relay must be refused"
);
config.node.allow_insecure = true;
let node = Node::start(config).await.unwrap();
let outcome = frxd::node::control_query(
&format!("http://{}", node.addr),
"rust",
Some(5),
Some(100),
false,
)
.await
.unwrap();
assert_eq!(
outcome.pointer("/local/total").and_then(Value::as_u64),
Some(0)
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn stale_envelopes_are_rejected() { async fn stale_envelopes_are_rejected() {
let relay_url = spawn_relay().await; let relay_url = spawn_relay().await;
+1
View File
@@ -38,6 +38,7 @@ fn save_registry(dir: &Path, ma: &Keypair, members: Vec<RegistryMember>) -> Path
version: 1, version: 1,
issued_at: now_ts(), issued_at: now_ts(),
ma_key: String::new(), ma_key: String::new(),
zone: "frx.invalid".to_string(),
members, members,
relays: Vec::new(), relays: Vec::new(),
}, },
+2 -2
View File
@@ -1,11 +1,10 @@
mod common; mod common;
use std::fs; use std::fs;
use std::path::Path;
use std::time::Duration; use std::time::Duration;
use common::{ use common::{
client, collection, config_for, messages_of_type, poll, poll_messages, publish, query_envelope, client, collection, config_for, messages_of_type, poll_messages, publish, query_envelope,
register, test_envelope, register, test_envelope,
}; };
use frxd::crypto::{Keypair, now_ts}; use frxd::crypto::{Keypair, now_ts};
@@ -99,6 +98,7 @@ fn registry_doc(
version, version,
issued_at: now_ts(), issued_at: now_ts(),
ma_key: String::new(), ma_key: String::new(),
zone: "frx.invalid".to_string(),
members, members,
relays, relays,
}, },
+142
View File
@@ -0,0 +1,142 @@
use std::path::Path;
use std::time::Duration;
use frxd::commands;
use frxd::config::Config;
use frxd::crypto::{Keypair, now_ts};
use frxd::onboard;
use frxd::registry::{self};
use serde_json::Value;
use tokio::net::TcpListener;
async fn spawn_registry_server(dir: &Path, signup_code: Option<&str>) -> String {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let base = format!("http://{addr}");
let router = commands::registry_router(
dir,
signup_code.map(str::to_string),
Some(format!("{base}/registry.json")),
);
tokio::spawn(async move {
let _ = axum::serve(listener, router).await;
});
base
}
fn setup_ma(root: &Path) -> std::path::PathBuf {
let dir = root.join("ma");
commands::registry_init(&dir, "frx.invalid").unwrap();
dir
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn signup_issues_invite_and_enroll_binds_key() {
let root = tempfile::tempdir().unwrap();
let dir = setup_ma(root.path());
let base = spawn_registry_server(&dir, Some("sesame")).await;
let http = reqwest::Client::builder()
.timeout(Duration::from_secs(5))
.build()
.unwrap();
let rejected = http
.post(format!("{base}/v1/signup"))
.json(&serde_json::json!({"label": "alice", "code": "wrong"}))
.send()
.await
.unwrap();
assert_eq!(rejected.status(), reqwest::StatusCode::FORBIDDEN);
let response = http
.post(format!("{base}/v1/signup"))
.json(&serde_json::json!({"label": "Alice Dev", "code": "sesame"}))
.send()
.await
.unwrap();
assert!(response.status().is_success());
let body: Value = response.json().await.unwrap();
let id = body.get("id").and_then(Value::as_str).unwrap().to_string();
assert_eq!(id, "alice-dev.frx.invalid");
let token = body
.get("token")
.and_then(Value::as_str)
.unwrap()
.to_string();
let signed = registry::load_registry(&dir.join("registry.json")).unwrap();
assert!(
registry::authorized_keys(&signed, now_ts()).is_empty(),
"signup must not authorize a key before enrollment"
);
let key = Keypair::generate();
let response = http
.post(format!("{base}/v1/enroll"))
.json(&serde_json::json!({
"id": id,
"token": token,
"pubkey": key.public_hex(),
}))
.send()
.await
.unwrap();
assert!(response.status().is_success());
let signed = registry::load_registry(&dir.join("registry.json")).unwrap();
assert!(registry::authorized_keys(&signed, now_ts()).contains_key(&key.public_hex()));
let replayed = http
.post(format!("{base}/v1/enroll"))
.json(&serde_json::json!({
"id": id,
"token": token,
"pubkey": Keypair::generate().public_hex(),
}))
.send()
.await
.unwrap();
assert_eq!(replayed.status(), reqwest::StatusCode::FORBIDDEN);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn wizard_enrolls_and_writes_config() {
let root = tempfile::tempdir().unwrap();
let dir = setup_ma(root.path());
let base = spawn_registry_server(&dir, Some("sesame")).await;
let http = reqwest::Client::new();
let body: Value = http
.post(format!("{base}/v1/signup"))
.json(&serde_json::json!({"label": "Wizard Test", "code": "sesame"}))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
let credentials = body.get("credentials").and_then(Value::as_str).unwrap();
let config_path = root.path().join("wizard.toml");
let input = format!("{}\n{credentials}\n\n\n\nn\n", config_path.display());
let mut reader = input.as_bytes();
let mut output = Vec::new();
onboard::run(&mut reader, &mut output, &config_path, None)
.await
.unwrap();
let text = String::from_utf8(output).unwrap();
assert!(text.contains("enrolled"), "{text}");
let config = Config::load(&config_path).unwrap();
assert_eq!(config.node.id.as_deref(), Some("wizard-test.frx.invalid"));
assert_eq!(
config.node.registry.as_deref(),
Some(format!("{base}/registry.json").as_str())
);
let key = config.load_key().unwrap();
let signed = registry::load_registry(&dir.join("registry.json")).unwrap();
assert!(
registry::authorized_keys(&signed, now_ts()).contains_key(&key.public_hex()),
"wizard did not bind the key"
);
}
+5 -4
View File
@@ -79,7 +79,7 @@ async fn no_in_protocol_citation_accounting_or_settlement() {
); );
} }
let response = serde_json::to_value(build_response("q1", Vec::new(), 0, 5)).unwrap(); let response = serde_json::to_value(build_response("q1", Vec::new(), Some(0), 5)).unwrap();
assert_absent_fields( assert_absent_fields(
&response, &response,
&[ &[
@@ -157,7 +157,7 @@ async fn no_protocol_query_dedup() {
#[test] #[test]
fn no_k_fetch_ingestion_attestations() { fn no_k_fetch_ingestion_attestations() {
let response = build_response("q1", Vec::new(), 0, 5); let response = build_response("q1", Vec::new(), Some(0), 5);
let value = serde_json::to_value(&response).unwrap(); let value = serde_json::to_value(&response).unwrap();
assert_exact_keys( assert_exact_keys(
&value, &value,
@@ -183,7 +183,7 @@ fn no_result_count_etiquette() {
&query, &query,
&["min_results", "results_count", "serp", "count_floor"], &["min_results", "results_count", "serp", "count_floor"],
); );
let response = serde_json::to_value(build_response("q1", Vec::new(), 0, 5)).unwrap(); let response = serde_json::to_value(build_response("q1", Vec::new(), Some(0), 5)).unwrap();
assert_absent_fields( assert_absent_fields(
&response, &response,
&["min_results", "results_count", "serp", "count_floor"], &["min_results", "results_count", "serp", "count_floor"],
@@ -212,7 +212,8 @@ fn no_global_reputation_score() {
"content", "content",
], ],
); );
let response_value = serde_json::to_value(build_response("q1", vec![item], 1, 5)).unwrap(); let response_value =
serde_json::to_value(build_response("q1", vec![item], Some(1), 5)).unwrap();
for value in [&item_value, &response_value] { for value in [&item_value, &response_value] {
assert_absent_fields(value, &["score", "rank", "reputation", "weight", "rating"]); assert_absent_fields(value, &["score", "rank", "reputation", "weight", "rating"]);
} }
+1
View File
@@ -41,6 +41,7 @@ fn ma_registry(ma: &Keypair, members: Vec<RegistryMember>, version: u64) -> Sign
version, version,
issued_at: now_ts(), issued_at: now_ts(),
ma_key: String::new(), ma_key: String::new(),
zone: "frx.invalid".to_string(),
members, members,
relays: Vec::new(), relays: Vec::new(),
}, },
+1 -1
View File
@@ -40,7 +40,7 @@ fn thousand_file_corpus_is_searchable_and_honest() {
assert_eq!(matches, total as u64); assert_eq!(matches, total as u64);
assert_eq!(hits.len(), 10); assert_eq!(hits.len(), 10);
let response = build_response("scale", response_items(&hits), matches, 10); let response = build_response("scale", response_items(&hits), Some(matches), 10);
assert!(response.truncated); assert!(response.truncated);
assert_eq!(response.more_available, (total - 10) as u64); assert_eq!(response.more_available, (total - 10) as u64);
assert!(serde_json::to_string(&response).unwrap().len() > 0); assert!(serde_json::to_string(&response).unwrap().len() > 0);