Freezing large indexes
Compact the full-text index of a large database into immutable segments that take several times less memory.
The full-text index of a database is built for fast inserts and removals: a radix tree of terms per string property, an array of document IDs per term and, per document, an object with the frequency of each of its terms. On large corpora those objects dominate memory: a 160,000-passage index takes 1.6 GB of heap, about 1 GB of it in per-document term frequencies, while its binary snapshot is 139 MB.
freeze(db) moves the full-text index into frozen segments: one immutable, columnar structure per string
property, with a sorted term dictionary and the document IDs and term frequencies of each term delta-encoded in
typed arrays.
import { create, insertMultiple, freeze, search } from "zbsearch";
const db = create({
schema: { title: "string", text: "string", year: "number" },
store: { exclude: ["text"] },
});
await insertMultiple(db, passages);
freeze(db);
search(db, { term: "brown fox", tolerance: 1 }); // same results, same scoresWhat changes, and what does not
Search results and scores do not change. Every search option works on frozen segments: prefix and exact
matching, tolerance, threshold, boost, properties, relevance, where filters (including filters on
string properties), facets, groupBy, sortBy, distinctOn, pinning rules, suggest, hybrid search, and the
corpus statistics of @zbsearch/sharded. Only the default index component
can be frozen: freeze throws FREEZE_NOT_SUPPORTED with a custom one (for example the QPS or PT15 plugins).
The database stays writable:
- Documents inserted after
freezego to the regular, mutable index (the tail), and searches read the frozen segments and the tail together. - Removing or updating a frozen document marks it as removed in its segment (a tombstone): its postings stay
in the segment, skipped by every search, until the next
freeze. BM25 statistics (document count, average field length, document frequency) are updated at once, exactly as for a mutable index. - The next
freeze(db)merges the segments, the tail and the tombstones into one new segment per property.
freeze takes time proportional to the size of the index (about 0.8 s for 20,000 passages and 6 to 8 s for
160,000 on the benchmark below) and blocks while it runs. Call it once a bulk load is done, or after
loading a snapshot of an unfrozen database.
Freezing automatically
To keep the tail small while documents keep coming in, give create a threshold:
const db = create({
schema,
freeze: { threshold: 10_000 },
});Once threshold documents have been inserted or removed since the last freeze (an update counts as one removal
and one insertion), the tail is frozen into a new segment, at the end of the insert, update or remove call
that reached it. Each automatic freeze only encodes the tail, so it costs time proportional to the tail, not to
the index; once there would be more than 8 segments per property, they are merged into one. Without freeze in
create, nothing is frozen unless you call freeze(db).
Saving and loading
save and saveBinary store the frozen segments, and load and loadBinary restore them as they are: loading a
frozen snapshot copies each typed array instead of rebuilding a radix tree and an object per document, so it is
much faster and the loaded database is frozen. A binary snapshot with frozen segments is written as format
version 2, which ZBSearch 4.0.x cannot read; a database without frozen segments is still written as version 1, as
before. Both versions are read.
Memory and speed
Synthetic passages (150-250 words from a Zipf-distributed 40,000-word vocabulary, string fields text, section,
title, itemKey, numbers page, year, enum collection) with store: { exclude: ["text"] }, Node 22, 4
vCPUs, 200 queries of 1-3 words per mode, measured with benchmarks/large-index.js and
benchmarks/memory-breakdown.js:
| 160,000 passages | 4.0.3 | Frozen |
|---|---|---|
| Heap | 1,634 MB | 220 MB |
| Search p50 / p95 | 110.6 / 212.5 ms | 8.2 / 16.5 ms |
Search with tolerance: 1, p50 / p95 | 205.0 / 411.3 ms | 19.1 / 36.0 ms |
| Update p50 | 106.2 ms | 0.16 ms |
Binary snapshot (saveBinary) | 138.7 MB, 8.6 s | 80.0 MB, 0.86 s |
loadBinary | 10.9 s | 0.82 s |
Most of the remaining heap is not the full-text index: of the 220 MB, the frozen segments take 62 MB, the sorter
90 MB and the documents store 38 MB. The sorter keeps every sortable value (string, number and boolean
properties) of every document; leave out the properties you never sort by with
sort: { unsortableProperties: [...] }, or disable sorting with sort: { enabled: false }.