Get started

How fyrr works

A writer, a reader and a bucket walk into a bar. Nobody brings a database.

In short

  • fyrr has two halves, a writer and a reader, that share nothing but an S3 prefix or a local directory.
  • This page defines the terms the other pages use and explains publication, authorization and retrieval.
  • Read it when you want to know what happens between a sync and a search result.

fyrr architecture

fyrr has two halves that share nothing but an S3 prefix (storage.uri), or on a single host a local directory:

Terms

TermMeaning
namespaceOne deployment's data under storage.uri, named by namespace.name. Every writer and reader of a deployment uses the same namespace.
sourceOne [[sources]] entry: a connector type, what it syncs (scope) and how its readers are decided (permissions).
reader tokenA user or group grant on a row: usr:<id> or grp:<id>. Stored only as an HMAC digest. Not to be confused with OIDC access tokens, which authenticate HTTP callers.
publicationOne immutable, published version of the namespace: the Lance content plus the policy and identity it is served with.
serving headThe single control object that names the current publication, policy and identity. Each commit replaces it in one conditional write.
pinThe publication one request reads from start to finish.
envelopeA durable batch a sync has validated and embedded, waiting for the committer to publish it.
mirrorThe reader's local cache of immutable published files (reader.mirror_path).
admissionThe reader's check, at the start of every request, that the caller is known and the pinned publication may be served.
LanceThe table storage and retrieval engine used for published text, vectors and indexes.
root_id / record_idA source's stable document ID / the ID of one text record within that document.
principal / canonical userThe deployment's user identity to which source accounts and authenticated subjects are linked.
committerThe writer holding the namespace lease while publishing prepared work.
incarnationA new identity assigned when a namespace is created, distinguishing it from earlier namespaces at the same URI.
freshness deadlineThe maximum age of a successful permission, membership or directory read before its grants stop being served (identity.validity_deadline).
quarantineWithholding an invalid document revision while retaining its last good published revision, if any.
providerA connector provider reads source content and access rights; a directory provider supplies canonical users and groups; an embeddings or reranker provider processes text for retrieval.

Publication

  1. Sync. fyrr writer sync runs each source in its own worker process under an expiring S3 lease per source and sync kind. Every batch is validated against the connector contract (Arrow schema, row rules, cursor chain, size bounds), embedded, and written as a durable envelope. The source's cursor advances only after its envelope is durable. An invalid or oversized document is quarantined and its last good revision stays visible.
  2. Commit. fyrr writer commit takes the namespace lease, which lives in the serving head; a second committer exits instead of waiting. The committer publishes permission and membership changes before content, appends rows and index deltas to the Lance dataset, writes an immutable content manifest and replaces the serving head with one conditional write (compare-and-swap on the object version). A failed conditional write leaves the writer refusing to continue (head_cas_failed).
  3. Serve. Each reader request pins one publication. Readers notice new publications in the background (reader.version_check_interval, 30 s, or SIGHUP) and switch between requests; in-flight requests keep their pin. A reader whose pin predates a published revocation stops serving until it has prepared the newer publication. Each namespace creation writes a new random incarnation into the serving head and every publication; a reader whose pin belongs to a namespace that was deleted and created again at the same storage URI stops serving at once and prepares the new namespace's publication.

Published versions are retained for writer.purge_window and can be restored; see operations.

Every artifact carries a major.minor contract version. An unknown major is refused with publication_invalid at startup or authorization_unavailable on a live request. A publication whose indexes or retrieval policy do not match is refused with incompatible_index. The exact formats are generated into schemas/ and fixtures/.

Local storage

A file:// root stores the same namespace objects and Lance dataset as S3. Each write stages its object in .fyrr-storage.stage.d/, flushes it, gives it its name with an atomic rename and flushes the directory before it reports success. An object's version is a random 128-bit tag in an extended attribute; an object without one is refused.

Writes to different objects never wait for each other. Creating an object takes no lock. Replacing or removing one (the serving head, a lease, a cursor) locks that object only for the comparison and the rename, never while flushing. A process stopped inside that step blocks later changes to that one object: they wait up to five seconds and then fail with storage_unavailable, head_cas_failed or sync_lease_lost, depending on the operation.

A reader can see an object before its writer has flushed the directory, so a writer that publishes a reference to something it only observed flushes it first. Lance uses an atomic create-exclusive link. Staging files of a killed writer are removed by later writes and by every delete; a stopped writer keeps its file. .fyrr-storage.lock remains in the root and stops earlier releases, which serialized all writes through it, from writing there. Maintenance and confirmed source purge reap abandoned multipart uploads (name#digits), protecting files younger than ten minutes and namespaces with live source leases; purge plans count them as namespace_staging_files. Reader mirrors copy immutable files and preserve in-flight pins across purge, bounded by reader.mirror_max_bytes.

Authorization

The writer replaces every reader token with an HMAC-SHA256 digest keyed by identity.hashing_key before it is stored. A published identity artifact maps each subject (the stdio user, or an OIDC subject over HTTP) to the hashed reader tokens it holds.

Every search, fetch and list_filters call passes three gates:

  1. Admission. The reader reads the live serving head, policy and identity. A malformed head, a head from another namespace, or missing or malformed policy or identity refuses with authorization_unavailable. An unreadable head or expired pin gives storage_unavailable on stdio. Admission also refuses a subject that is not in the identity artifact or holds more than identity.max_tokens_per_user reader tokens, and an expired directory read (identity.validity_deadline) when a directory provider is configured. A permissions = "sync" source whose last successful permission or membership read is older than identity.validity_deadline is excluded, and named in metadata.hidden_sources if the caller could otherwise read it. A retired source is excluded silently.
  2. Permission predicate. Lexical and dense candidates are generated under a filter on the row's hashed readers, so a row the caller may not read is never scored, counted or returned. For permissions = "fixed" sources the filter checks the readers in the current policy, not the stored rows.
  3. Final check. Before a result or error is returned, the reader reads the serving head again. If the policy, identity or hidden sources changed during the request, the request is refused rather than answered under the old grant.

Unknown and denied documents look the same to fetch: an empty result. Source revocations require a successful sync that reads the changed rights and a commit that publishes them; in-run changes can be missed.

Retrieval

A search runs two candidate lanes over the permission-filtered rows:

The two score lists are normalized, fused (weight 0.5 lexical, 0.5 dense by default) and grouped to one hit per document. A soft per-source cap spreads the top of the list across sources; remaining slots may still come from the same source. An optional reranker ([namespace.rerank]) reorders the authorized documents. Excerpts are passages of up to 1200 codepoints around the query terms; fetch returns whole documents or a window of one record.

Fusion weights, depths, analyzer settings and composition enter one retrieval-policy digest that is published with each version, so readers and writers cannot silently disagree about ranking. The tuning keys are in tuning; the tool arguments and outputs are in MCP tools.

Edit this page on GitHub ↗