Raise matching floor: boundary tokenizer, stemming/folding/stopwords, boosts, snippets, engine seam

This commit is contained in:
George Coles
2026-09-15 08:11:55 -04:00
parent 0d35036168
commit 382acc36a9
10 changed files with 294 additions and 42 deletions
+4 -1
View File
@@ -86,6 +86,8 @@ Decisions taken during design review, with rationale.
| 15 | Centralize coordination, localize judgment (I2) | Common state is cheaper held once: identity, admission, contract in the MA; matching, relevance, sharing, retention local | I2 reframed |
| 16 | No sessions; per-message signatures | Peers are not connected; mailbox auth is a transport-local proof of possession | Implemented |
| 17 | Lexical coverage gate before any rerank | Precision is project health; a demo false positive showed raw OR matching is too weak; embeddings later, local and replaceable | Implemented (`[match] min_coverage`) |
| 18 | Default engine: boundary tokenizer + fold + stopwords + stemmer, title/phrase boosts, query-time snippets | The floor must be high out of the box; model-number and morphology matching are cheap wins with no model | Implemented (`src/tokenizer.rs`) |
| 19 | Engine seam: `SearchEngine` trait with the responder path as the conformance wrapper | Plugins can change quality, never conformance; engines return `Option<total>` so an external engine can't fake the truncation bit | Implemented (`src/engine.rs`) |
## 8. Implementation status
@@ -93,7 +95,8 @@ Built and tested (99 tests):
- Envelope, JCS signing, registry binding, freshness window
- Tantivy index, collections manifest, shared/exposure enforcement, reindex reset
- Coverage-gated lexical matching
- Coverage-gated lexical matching with boundary tokenization, stemming, folding, boosts, and snippets
- Engine seam (`SearchEngine` trait) behind the conformance wrapper
- Query broadcast, SSE streaming, long-poll fallback, per-member queues and lag reporting
- Relay federation, relay admission, registry watcher (path/URL, monotonic, fail-static)
- Registry CLI (init/add/add-key/revoke-key/set-enc-key/set-relays/show/serve), key rotation