The open comparison / At a glance

Search, compared.

Similar search quality. A fraction of the estimated running cost. Faster queries and indexing.

Compared where measured. Scope & caveats ↘

Quality: the right documents in the top 10. Services: always-on services the product ships and requires.

SystemMonthly costServicesMemoryQuery speedIngestQuality
fyrr†
Monthly cost$42†Estimate / month
Services1†reader; writer = scheduled job; plus S3
Memory92.1 MiB†At idle
Query speed263 ms†Repeated: 15 ms
Ingest11.2 min†1,791 docs/min
Quality94.8 %†Recall@10
Onyx†
Monthly cost$858†Estimate / month
Services12†Managed extras
Memory9.1 GiB†At idle
Query speed423 ms†Repeated: 78 ms
Ingest16.6 min†1,206 docs/min
Quality94.5 %†Recall@10
PipesHub†
Monthly cost≥ $2,552†Known subtotal / month
Services15†Managed extras
Memory9.5 GiB†At idle
Query speed920 ms†Repeated: 662 ms
Ingest~71–87 docs/min†Segments; full time n/e
Quality89.5 %†Recall@10

PipesHub: non-default config, partial coverage — details ↘n/e = not evaluated · Scope & method ↘

EnterpriseRAG-derived · gold document relevanceOpen protocol. Every attempted system.
Customize systems & metrics · copy selection link
Systems

Overview: default fyrr and Onyx; PipesHub uses source-documented, non-default parallel partition scheduling and owner-selected gpt-6-luna for native enrichment. Its retrieval and latency are measured with 19,998 / 20,000 documents searchable, two pending duplicates and background enrichment active; canonical ingest time remains n/e. Optional details: fyrr reader.concurrency=8 on the same index, and two experimental AWS Lambda deployments of the real 20k corpus (4096 MiB and 2048 MiB). Neither changes headline wins. Lambda cold starts remain in their timings; their complete operating cost is n/e. The older 300-document synthetic trial is a separate projection. Neither real-20k Lambda deployment met the predeclared C8 latency selector; no Lambda winner is selected.

Choose metrics
Metrics

Bars show absolute values. Scores show the percentage-point gap to the best selected system; time, cost and footprint use ratios (×1.00 best). Incomplete or missing measurement samples are unranked; input coverage is disclosed separately.

01 / Footprint & cost

Costs a fraction to run.

Estimated monthly cost on AWS Frankfurt, with the same reference workload.

Footprint & costlower is better

Estimated production cost

USD/month

fyrr† $42best
Onyx† $858×20.30
PipesHub† n/en/e comparison

95% intervals · measurement notes ↘

Footprint & cost details ↘

02 / Search latency

Answers fast.

New questions include query embedding; repeated questions preserve each product’s native caching.

Search latencylower is better

New question

ms

p50 bar p95 tickWhiskers: 95% CI

fyrr† p50 263p95 515best
Onyx† p50 423p95 916×1.61
PipesHub† p50 920p95 2,715×3.50

95% intervals · measurement notes ↘

Search latency / secondarylower is better

Repeated question

ms

p50 bar p95 tickWhiskers: 95% CI

fyrr† p50 15p95 18best
Onyx† p50 78p95 90×5.19
PipesHub† p50 662p95 1,381×44.05

95% intervals · measurement notes ↘

Search latency details ↘

03 / Ingest & embeddings

Indexes fast.

Wall time to make accepted documents searchable, from the same 20,000-document submission.

Ingest & embeddingslower is better

Time to searchable index

seconds

fyrr† 670 s1,790.7 docs/minbest
Onyx† 995 s1,205.88 docs/min×1.49
PipesHub† n/en/e docs/minn/e comparison

95% intervals · measurement notes ↘

Ingest & embeddings details ↘

04 / Retrieval quality

Finds the right documents (comparable).

Recall@10 measures whether the right documents appear in the first ten results.

Retrieval qualityhigher is better

Recall@10

Score · full scale 0–100%

fyrr† 94.8%best
Onyx† 94.5%-0.3 pp
PipesHub† 89.5%-5.3 pp

95% intervals · measurement notes ↘

Retrieval quality details ↘

05 / The trade-offs

Choose what you
want to operate.

A retrieval engine and a full AI workspace solve different problems. The benchmark is one part of the decision.

fyrr

Permission-aware retrieval over MCP

Permission-aware search over index files on S3; no database or cluster to run.

Returns evidence to your agent; does not supply an answer-generation UI.

Strengths, limits & run notes

Run version: 0.1.0

Where it fits

  • Permission-aware search over index files on S3; no database or cluster to run.
  • Choose an OpenAI-compatible embedding endpoint and an optional configured reranker.
  • No answer-model API key in fyrr; your existing MCP app writes the answer.

Trade-offs & limits

  • Returns evidence to your agent; does not supply an answer-generation UI.
  • Cloud connectors have only been tested against synthetic APIs, not real workspaces.
  • Readers still need compute, a private disk mirror and an endpoint for live query embeddings.

19,999 / 20,000 accepted · 1 rejected · 19,999 / 20,000 searchable · 0 pending

New-question C8: 4 / 250 valid; 246 missing or refused. Incomplete percentiles remain unranked.

Ingest coverage 19,999/20,000; 1 rejected document(s), with native reasons and IDs in ingest.json. No gold questions excluded.

Architecture & deployment ↗ · Connector maturity ↗ · Model configuration ↗ · Reranker configuration ↗ · Existing AI app ↗

Run JSON: version, date & config ↗

Onyx

AI workspace with search & agents

Built-in chat, custom agents and deep research.

Self-hosting the full search deployment means operating its application and data services.

Strengths, limits & run notes

Run version: 5.1.2

Where it fits

  • Built-in chat, custom agents and deep research.
  • A broad connector catalog and self-hosted deployment options.

Trade-offs & limits

  • Self-hosting the full search deployment means operating its application and data services.
  • Its full workspace includes features beyond this retrieval-only test.
  • The measured local stack has 12 resident services. The production cost model prices its documented cloud deployment, which has different availability and topology from the local benchmark.

20,000 / 20,000 accepted · 0 rejected · 20,000 / 20,000 searchable · 0 pending

New-question C8: 250 / 250 valid; 0 missing or refused. Incomplete percentiles remain unranked.

Core MIT; ee paths under the Onyx Enterprise License. Standard released topology: 12 resident containers, plus on-demand executor image dependency.

Rootless-host adaptations: bootstrap.memory_lock=false; default 2GiB OpenSearch heap; absolute disk watermarks low20gb/high10gb/flood5gb. Main OpenSearch JVM VmSwap stayed zero during ingest and both latency blocks.

All 20000 exact input files accepted. Default product parser removed the NUL character, confirmed by native chunk API; adapter submitted unchanged bytes. Native rejected count0. No gold exclusions.

Ordinary first-admin session on public Search UI/File connector endpoints; service-account API keys require Business tier in v5.1.2. Primary bulk path is File connector with deterministic ZIP order, normal background defaults.

Actual Voyage HTTP attempts/tokens/cost n/e; native embedding batch events are not provider-attempt counts. Held budget reservations are not billed cost.

Product & deployment source ↗ · Deployment services ↗

Run JSON: version, date & config ↗

PipesHub

Enterprise search & context platform

Cited answers, knowledge-graph navigation and agent tools.

Self-hosting includes vector, graph and application data stores, alongside the application services.

Strengths, limits & run notes

Run version: 1.0.0

Where it fits

  • Cited answers, knowledge-graph navigation and agent tools.
  • Connects business sources and exposes context through MCP.

Trade-offs & limits

  • Self-hosting includes vector, graph and application data stores, alongside the application services.
  • Its agent features are outside this retrieval and search-latency comparison.
  • The measured non-default configuration kept background enrichment active, with two of 20,000 accepted documents still pending. Search quality and latency are measured; canonical ingest time and complete operating cost remain n/e. Sample rates and infrastructure subtotals are qualified separately.

20,000 / 20,000 accepted · 0 rejected · 19,998 / 20,000 searchable · 2 pending

New-question C8: 250 / 250 valid; 0 missing or refused. Incomplete percentiles remain unranked.

FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and… Configuration source ↗Partial searchability: 99.99% searchable versus 100.0% accepted (19,998 / 20,000 submitted; 2 pending). Search measured at this coverage.Background document enrichment active during search measurements. Enrichment completed: 5,848 → 7,236; pending: 14,152 → 12,764; failed: 0 → 0.

FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and explicitly projected full time. Partial searchable coverage 19,998/20,000; 20,000 accepted, 2 pending duplicate copies awaiting twin classification. Native completed-ID inventory and late probe verified; no scoring exclusions. Canonical ingest time/throughput n/e after a harness poller caused a stop. Segment rates and gap retained separately. Native document enrichment active during measurement. Observed start/end completed 5848/7236, pending 14152/12764, failed 0/0 (per-phase records retained).

FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and explicitly projected full time. Partial searchable coverage 19,998/20,000; 20,000 accepted, 2 pending duplicate copies awaiting twin classification. Native completed-ID inventory and late probe verified; no scoring exclusions. Canonical ingest time/throughput n/e after a harness poller caused a stop. Segment rates and gap retained separately. Native document enrichment active during measurement. Observed start/end completed 5848/7236, pending 14152/12764, failed 0/0 (per-phase records retained).

Partial searchable coverage verified: 19,998/20,000 searchable, 20,000 accepted, 2 named pending document(s). All question/gold denominators remain unchanged. Canonical ingest time/throughput are n/e because the harness stopped the product; gap and segment rates remain diagnostic only.

Current-index measurements authorized with explicit partial searchable coverage; never claim every accepted record searchable.

No pending copy or its twin is gold for any selected250question. All questions/golds retained unchanged.

Native document enrichment remains active during measurements; per-phase boundary counts retained.

Original failed ingest preserved; canonical ingest wall/docs-min n/e.

After a10min two-record tail, normal documented UI/API reindex was requested once for each existing queued record; both eventPublished=true. No files reuploaded, source/config unchanged. Subsequent diagnosis verified duplicate classification dependency.

Product & deployment source ↗ · System architecture ↗

Run JSON: version, date & config ↗

All ingest diagnostics ↘

Model choice & feature facts

Bring your own models

Configuration support is separate from benchmark performance. Sources and release-specific limits are shown for every product.

Model choicefyrrOnyxPipesHub
Embedding modelChoose the endpoint and model in [embedding]. The provider must implement the OpenAI-compatible embeddings protocol.Configuration ↗Self-hosted installs choose from a local/cloud model registry. In v5.1.2, new Voyage selections use the voyage-4 family.Release model registry ↗ · Index settings ↗Choose a supported embedding provider and model, including local or cloud models; Voyage is supported.AI models ↗
Reranker on the tested search pathOptional endpoint and model in [namespace.rerank], with candidate, deadline and spend limits. No reranker calls when absent.Reranker settings ↗v5.1.2: the selected Search UI endpoint uses native hybrid ranking without a neural reranker. Unused reranker helpers do not establish support on this path.Release search pipeline ↗v1.0.0: /api/v1/retrieve retains native hybrid/RRF ranking. Its request exposes no hosted reranker selection; the local reranker helper is not invoked on this path.Release retrieval endpoint ↗
Answer model & API keyNo answer-LLM API key is configured in fyrr. Search and fetch supply evidence to your existing MCP app; its plan and limits still apply. Embeddings and optional reranking may need separate credentials.Claude Code example ↗ · MCP tools ↗Built-in answers use a configured LLM. External AI apps can also access Onyx through MCP.MCP access ↗Built-in answers use the configured language model. MCP also exposes context to external agents.System overview ↗ · MCP overview ↗
Programmatic accessStdio for a local client; Streamable HTTP authenticated with OIDC for shared access.MCP transports ↗ · HTTP users ↗Service-account API keys require Business, Enterprise or a paid Cloud plan. User tokens/session auth are separate options; the benchmark uses a normal user session.v5.1.2 authentication ↗MCP supports personal access tokens or OAuth. Access follows the authenticated user and their permissions.MCP authentication ↗

06 / Show the working

Cost assumptions & deployment choices
Experimental Lambda trial & projection
Search latency, caching & concurrency 8

fyrr default C8: 4 / 250 valid responses; 246 missing or refused. Incomplete percentiles are unranked; reader.concurrency=8 is a separate optional run.

New question includes query embedding. Repeated question asks the same query again, preserving native cache behavior.fyrr: in-process, reset on restart; Onyx: Redis tenant/settings/query hash TTL900s, ephemeral Redis restarted; PipesHub: No cross-request query-vector cache found in selected search path

Search latencylower is better

New question

ms

p50 bar p95 tickWhiskers: 95% CI

fyrr p50 263.17p95 514.71best
Onyx p50 423.1p95 915.69×1.61
PipesHub · non-default p50 920.3p95 2,715.31×3.50 FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and… Configuration source ↗Partial searchability: 99.99% searchable versus 100.0% accepted (19,998 / 20,000 submitted; 2 pending). Search measured at this coverage.Background document enrichment active during search measurements. Enrichment completed: 5,848 → 7,236; pending: 14,152 → 12,764; failed: 0 → 0.

Whiskers show 95% confidence intervals where available. Incomplete measurement samples cannot establish a product win.

Samples, uncertainty & notes
  • fyrr · First-ask p50 · concurrency 1: measured · valid 250 / attempted 250 · missing 0 · 95% CI 261.51–265.09 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/fyrr-release-full/latency.jsonl
  • fyrr · p95: measured · valid 250 / attempted 250 · missing 0 · 95% CI 327.68–624.13 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/fyrr-release-full/latency.jsonl
  • Onyx · First-ask p50 · concurrency 1: measured · valid 250 / attempted 250 · missing 0 · 95% CI 415.52–438.73 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/onyx-full/latency.jsonl
  • Onyx · p95: measured · valid 250 / attempted 250 · missing 0 · 95% CI 898.05–959.59 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/onyx-full/latency.jsonl
  • PipesHub · non-default · First-ask p50 · concurrency 1: measured · valid 250 / attempted 250 · missing 0 · 95% CI 841.79–1,079.1 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/pipeshub-full-final-clean/latency.jsonl
  • PipesHub · non-default · p95: measured · valid 250 / attempted 250 · missing 0 · 95% CI 2,158.29–3,239.79 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/pipeshub-full-final-clean/latency.jsonl
Search latency / secondarylower is better

Repeated question

ms

p50 bar p95 tickWhiskers: 95% CI

fyrr p50 15.03p95 18.43best
Onyx p50 78.09p95 89.65×5.19
PipesHub · non-default p50 662.12p95 1,380.67×44.05 FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and… Configuration source ↗Partial searchability: 99.99% searchable versus 100.0% accepted (19,998 / 20,000 submitted; 2 pending). Search measured at this coverage.Background document enrichment active during search measurements. Enrichment completed: 5,848 → 7,236; pending: 14,152 → 12,764; failed: 0 → 0.

Whiskers show 95% confidence intervals where available. Incomplete measurement samples cannot establish a product win.

Samples, uncertainty & notes
  • fyrr · Warm search C1 p50: measured · valid 250 / attempted 250 · missing 0 · 95% CI 14.85–15.53 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/fyrr-release-full/latency.jsonl
  • fyrr · p95: measured · valid 250 / attempted 250 · missing 0 · 95% CI 17.84–19.3 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/fyrr-release-full/latency.jsonl
  • Onyx · Warm search C1 p50: measured · valid 250 / attempted 250 · missing 0 · 95% CI 77.35–79.25 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/onyx-full/latency.jsonl
  • Onyx · p95: measured · valid 250 / attempted 250 · missing 0 · 95% CI 87.31–90.63 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/onyx-full/latency.jsonl
  • PipesHub · non-default · Warm search C1 p50: measured · valid 250 / attempted 250 · missing 0 · 95% CI 640.24–690.06 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/pipeshub-full-final-clean/latency.jsonl
  • PipesHub · non-default · p95: measured · valid 250 / attempted 250 · missing 0 · 95% CI 1,247.32–1,621.79 · One request per question, valid responses only; percentile bootstrap 10000 seed1729; failures reported separately · runtime/pipeshub-full-final-clean/latency.jsonl
Ingest stages, embedding costs & diagnostic samples
Ingest & embeddingslower is better

Time to searchable index

seconds

20,000 documents submitted · wall time until every accepted document is searchable. Rejections are disclosed per product. Stacks show disjoint elapsed stages; unreported time and native rejection reasons are in details.

fyrr 670.1 s1,790.7 docs/minbest 19,999 / 20,000 accepted · 1 rejected · 19,999 / 20,000 searchable · 0 pending Embedding cost: n/e
Onyx 995.12 s1,205.88 docs/min×1.49 20,000 / 20,000 accepted · 0 rejected · 20,000 / 20,000 searchable · 0 pending Embedding cost: n/e
PipesHub · non-default n/en/e docs/minn/e comparison FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and… Configuration source ↗Partial searchability: 99.99% searchable versus 100.0% accepted (19,998 / 20,000 submitted; 2 pending). Search measured at this coverage.Background document enrichment active during search measurements. Enrichment completed: 5,848 → 7,236; pending: 14,152 → 12,764; failed: 0 → 0.
Full configuration & interruption notes

FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and explicitly projected full time. Partial searchable coverage 19,998/20,000; 20,000 accepted, 2 pending duplicate copies awaiting twin classification. Native completed-ID inventory and late probe verified; no scoring exclusions. Canonical ingest time/throughput n/e after a harness poller caused a stop. Segment rates and gap retained separately. Native document enrichment active during measurement. Observed start/end completed 5848/7236, pending 14152/12764, failed 0/0 (per-phase records retained).

FAIR_SCHEDULING_PARALLEL_PARTITIONS=true, non-default, source-documented; indexing LLM gpt-6-luna (owner choice); default 20,000-document ingest not completed. See default_ingest_baseline for measured sample rate and explicitly projected full time. Partial searchable coverage 19,998/20,000; 20,000 accepted, 2 pending duplicate copies awaiting twin classification. Native completed-ID inventory and late probe verified; no scoring exclusions. Canonical ingest time/throughput n/e after a harness poller caused a stop. Segment rates and gap retained separately. Native document enrichment active during measurement. Observed start/end completed 5848/7236, pending 14152/12764, failed 0/0 (per-phase records retained).

Partial searchable coverage verified: 19,998/20,000 searchable, 20,000 accepted, 2 named pending document(s). All question/gold denominators remain unchanged. Canonical ingest time/throughput are n/e because the harness stopped the product; gap and segment rates remain diagnostic only.

Current-index measurements authorized with explicit partial searchable coverage; never claim every accepted record searchable.

No pending copy or its twin is gold for any selected250question. All questions/golds retained unchanged.

Native document enrichment remains active during measurements; per-phase boundary counts retained.

Original failed ingest preserved; canonical ingest wall/docs-min n/e.

After a10min two-record tail, normal documented UI/API reindex was requested once for each existing queued record; both eventPublished=true. No files reuploaded, source/config unchanged. Subsequent diagnosis verified duplicate classification dependency.

20,000 / 20,000 accepted · 0 rejected · 19,998 / 20,000 searchable · 2 pending Embedding cost: n/e

Whiskers show 95% confidence intervals where available. Incomplete measurement samples cannot establish a product win.

Samples, uncertainty & notes
  • fyrr · Time to searchable index: measured · valid 1 / attempted 1 · missing 0 · 95% CI n/e · runtime/fyrr-release-full/ingest.json
  • fyrr · writer sync: 663.63 s · writer commit: 4.14 s · reader check: 0.27 s · mcp inventory readiness: 0.3 s · late document probe: 1.76 s · Other / unreported: 0.0015 s
  • Onyx · Time to searchable index: measured · valid 1 / attempted 1 · missing 0 · 95% CI n/e · runtime/onyx-full/ingest.json
  • Onyx · Stage breakdown n/e; total wall time only.
  • PipesHub · non-default · Time to searchable index: n/e · valid 0 / attempted 1 · missing 1 · 95% CI n/e · runtime/pipeshub-full-final-clean/ingest.json
  • PipesHub · non-default · Stage breakdown n/e; total wall time only.
Question types, individual questions & all metrics

The detailed notebook loads when opened.

Methodology, provenance & downloads

Compared where measured. PipesHub: ingest time n/e, total cost n/e; 19,998 / 20,000 searchable (2 pending); see notes.

Status & scope
Measured results, approved for publication. Fyrr and Onyx completed their accepted-document ingest protocols. PipesHub retrieval and latency use 19,998 searchable of 20,000 accepted documents, with two duplicate copies pending; all 250 questions and original gold labels are retained. Its canonical ingest clock is unavailable after a harness stop/resume, and its full production cost is n/e. This synthetic English subset has incomplete relevance labels and reduced context; it cannot establish production performance.
Subset derivation
Frozen: 250 of the core 500 questions, stratified by official type using Hamilton allocation and stable SHA-256 selection. All selected gold documents plus source-proportional distractors make 20,000 documents. 379 gold documents; 96,402,403 corpus bytes. Seed: fyrr-compare-v1-2026-11-16. The separate 10-question / 1,000-document smoke test is never a published benchmark. Exact freeze and protocol hashes are recorded below.
Embeddings
Voyage voyage-4 with 1,024 dimensions for the measured systems, using document/query input types. Native chunking, request prefixes, deduplication and caches are retained and recorded per run. Native provider-billed embedding spend is n/e where unavailable; canonical-input API cost estimates are separate.
Uncertainty & missing results
95% percentile bootstrap intervals: 10,000 resamples, seed 1729, paired question IDs. Expected is the frozen denominator; attempted counts actual attempts; n counts valid results; missing = expected minus n, including unattempted rows. Retrieval excludes 15 empty-gold questions, leaving 235. Incomplete aggregates are partial. Unequal successful subsets cannot establish a product win.
Speed protocol
Primary: ask each of the 250 questions once per fresh service at concurrency 1 and 8, after one unrelated index warm-up. Fresh query caches; embedding-provider time included; percentile bootstrap intervals. Repeated-query warm timings remain a separate secondary track. Timed runs require a quiet host.
Hardware, versions & measurement date
Measured 17 November 2026. Native host: Intel Core i9-12900H, 14 physical cores / 20 logical CPUs, 30.576 GiB kernel-reported RAM, Linux 7.0.0-42-generic / glibc 2.43. Captured CPU, memory and kernel invariants match the native fyrr and Onyx windows. Experimental Lambda readers ran separately on AWS eu-central-1, x86_64, at 4096 or 2048 MiB. Exact commits, build/configuration hashes, phase windows and hardware-source hashes are in the JSON provenance.
Cost assumptions
Main monthly cost prices each product’s documented production deployment on AWS Frankfurt for 20,000 documents, 100,000 searches per month and 1% changed daily. Required replicas and managed services are itemized with deployment and price sources. Unknown required costs leave the total n/e, with a known subtotal shown separately. The measured single-host minimum uses matched light/C1 and team/C8 capacity references; the Lambda trial is an independent projection. These price models do not establish an AWS capacity or availability guarantee. Service counts exclude scheduled writer jobs and generic TLS termination; the cost model may still budget for them.
Headline wording
Each headline clause compares all representative products with complete measured values for that metric. n/e and partial rows cannot establish a win and are disclosed separately. Similar search quality uses at most a 2 pp gap or overlapping 95% intervals for recall@10 and nDCG@10 when both are measured; this is a descriptive rule, not an equivalence test. Faster queries require separated C1 median intervals; indexing uses accepted-document wall time. A fraction of cost requires complete stated production models and at most half the least expensive measured alternative. Overview numbers are rounded; details and JSON retain full precision.
Reproducibility artifacts
Raw evidence bundle (43.6 MB, SHA-256 dd0bf0f454acb5fc617a716cbc819eca6e85ef5639e6f6ee1d2c4fd26bfdca66) available on request; public hosting pending. Kept in a private fyrr-ai repository. Independent replay of the archived scorer reproduced all 30 aggregate means and sample counts (n=235 across five quality rows).
Service count definition
The Services column counts always-on services the product ships and requires: fyrr 1 reader, Onyx 12, PipesHub 15. The fyrr writer runs as a scheduled job; its index is on S3. Generic TLS termination and managed infrastructure are listed separately, and the TLS-proxy cost allowance is not counted as a fyrr service. Raw local resident-process counts remain in the run JSON; the documented production PipesHub profile differs from its measured local stack.
Versions, dates, configurations & provenance

Full provenance loads when opened. Download results JSON ↗

EnterpriseRAG dataset attribution · MIT license
MIT License

Copyright (c) 2026 DanswerAI, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Dataset source ↗
Add your system / reproduce a run
  1. Use the frozen corpus and question selection; verify the manifest hashes.
  2. Implement the benchmark adapter against the actual product API. Preserve native ranking and return product-selected evidence.
  3. Run the shared protocol. Record every attempt, missing result, exact version, date, hardware and nonsecret configuration.
  4. Export one schema 1.1 results JSON per run, including the metric catalogue, confidence intervals, artifacts and detailed rows.
  5. Use the downloadable JSON to inspect each run. The benchmark source and raw evidence are kept in a private fyrr-ai repository; request access to reproduce a run or contribute another system. Public hosting is pending.

Published by the fyrr team. Product descriptions link to their sources; measurements apply only to the exact configurations recorded above. Comparison repository: kept in a private fyrr-ai repository; raw evidence available on request, public hosting pending.

More products and metrics coming. Every caveat, configuration and raw result stays in the notebook.

Get started

Chart your
own islands.

$ git clone https://github.com/fyrr-ai/fyrr && cd fyrr
$ scripts/doctor --local
$ examples/quickstart/run

Pre-release: builds from source on Linux; the build prerequisites (including openssl and jq) are listed in CONTRIBUTING. Cloud connectors are tested against synthetic APIs so far.