These runs use different corpus sizes, models, configurations and sampling windows. Observed rates and linear projections are diagnostics, not completed 20,000-document results or a controlled comparison. They do not enter the ingest bars or headline.
Sampling, configuration & limitations
Default
Measured from an interrupted 1,000-file default smoke, Docker project cmp-pipeshub. All 1,000 original files uploaded unchanged; neither the 1,000-file nor the 20,000-file ingest completed. Canonical full-ingest time and rate remain n/e.
Two approved active sampling segments: 2026-11-17T03:43:45.135889+00:00 to 2026-11-17T04:08:07.575520+00:00 (33 to 218 completed, 185 new); 2026-11-17T04:18:17.619683+00:00 to 2026-11-17T04:24:01.899613+00:00 (231 to 281 completed, 50 new). Total: 235 new documents over 1,806.720 s. The 610.044 s reload/startup gap and 13 gap completions are excluded.
Default Kafka record-events processing serializes one record per partition; this KB shares one partition. Worker/index admission counts alone do not parallelize that partition. The later parallel-partitions row changes this nondefault source-documented setting.
Native default indexing behavior includes asynchronous LLM classification and entity resolution, configured here with OpenAI gpt-4o-mini; named-entity extraction/high-accuracy mode stayed off. Classification can continue after searchability.
An approved transparent OpenAI usage/budget proxy forwards the same model and raw bodies/responses. Cached entity-resolution clients retained the old URL after live configuration, so the app was reloaded once to ensure all subsequent model calls pass through the $15 cap. All post-reload native LLM admissions matched observed proxy requests. Prior unknown exposure remains held; exact prior billing n/e.
The proxy is one separate measurement container on the private app network, with its host port bound to loopback; excluded from the eight product services. Local RPC median 0.67 ms; initial forwarding overhead 2–5 ms on 3–4 s LLM calls; full per-call diagnostics are retained. Instrumentation overhead is not subtracted from the measured rate.
Projection is linear 20,000/rate, not verified completion. It assumes unchanged rate and ignores startup/upload/transient effects and corpus-scale/source variation; early sorted source order is dominated by Gmail, whereas the full corpus has a different mix. This diagnostic is not comparable to completed 20k ingest speed.
Zero observed native indexing FAILED records at both segment boundaries; unprocessed records are pending, not rejected. A shutdown-related transient failure is retained separately as operator interruption, not a document rejection.
Full diagnostic configuration and observations ↗
Parallel partitions
Diagnostic rate from owner-interrupted20k project cmp-pipeshub-full-luna: native COMPLETED inventory delta1311 (26to1337) over1518.281806s, from2026-11-17T06:08:19.303400Z to2026-11-17T06:33:37.585206Z.
Both rate endpoints are timestamped native inventory observations; no unmatched harness-start time or inferred first-submission clock is used. Initial26 completions and time before first observation are excluded.
This is a partial-import progress rate, including overlapping upload/index/enrichment work, not completed20k ingestion speed or necessarily steady-state throughput. The early common sorted source cohort differs from the final corpus mix.
Non-default parallel partitions true; owner-selected Luna model; native classification and taxonomy resolution active. Proxy instrumentation included in elapsed time, no overhead subtraction.
All20k documents submitted unchanged, but searchable completion/native accepted inventory/shared late probe not reached. Last pre-stop observation1337complete,35indexing,18628queued,0indexing failures. Pending is not rejected.
PROJECTED20k time is linear20000/rate; it is not verified completion and ignores source/length/scale/provider/queue variation. Default baseline uses a different explicitly disclosed partial sampling window; neither is a canonical completed20k result.
No supported enrichment-off classification/taxonomy-resolution switch in v1.0.0; no upstream code patch or enrichment-off rerun.
Full diagnostic configuration and observations ↗
Scaled indexing
Measured completed1k rate only, published as a labelled diagnostic by owner487be118. It is not canonical20k ingestion speed, retrieval quality or latency.
Exact1000 native COMPLETED inventory and shared late-document probe verified; original bytes/UI upload defaults retained. No document rejected.
PROJECTED20k time is a linear extrapolation from1k. Dataset source/length/scale/provider/scheduling variation means it is not a verified full duration.
This profile did not improve the observed smoke rate and misses the2.5h condition. Owner stopped further experiments; no scaled full20k run started.
Default, parallel and scaled diagnostics have different disclosed sampling windows/corpus sizes; do not infer a controlled speedup or compare these rates directly with completed20k canonical ingest.
Higher worker/concurrency profile increased observed memory and product cgroup swap (peak3,392,196,608bytes), although MemAvailable stayed above8GiB. This is an explicit host/memory caveat, not a causal decomposition of the slowdown.
Native enrichment remained enabled: snapshot900 completed/100 queued/0failed, before final shutdown. Its unfinished tail and unknown provider usage are preserved.
All owned services/proxy stopped; volumes retained. Retrieval/latency/full20k query footprint and capacity remain n/e for PipesHub.
Full diagnostic configuration and observations ↗
Full final segment1
Segment rates use matched native count timestamps; no projection is completed20k ingest timing.
Resume window contains10min duplicate tail and native reindex recovery; not a steady-state throughput claim.
First segment includes15.076s proxy-instrumentation pause; stop gaps and startup contention separately retained.
These two reopened full-run observation segments are distinct from earlier default/parallel/scaled diagnostics; none is a completed20k ingest time.
Full diagnostic configuration and observations ↗
Full final segment2
Segment rates use matched native count timestamps; no projection is completed20k ingest timing.
Resume window contains10min duplicate tail and native reindex recovery; not a steady-state throughput claim.
First segment includes15.076s proxy-instrumentation pause; stop gaps and startup contention separately retained.
These two reopened full-run observation segments are distinct from earlier default/parallel/scaled diagnostics; none is a completed20k ingest time.
Full diagnostic configuration and observations ↗