Indexing & rebuilds

ZBSearch Edge separates writes from snapshot promotion on purpose. Search loads the live snapshot and, when pendingOps > 0, overlays pending WAL operations in memory so recent writes can appear before the next rebuild completes.

Write path: the WAL

When you upsert or delete a document, the Worker appends the operation to an append-only WAL in R2. How the append happens depends on whether the IndexCoordinator Durable Object binding is present (it is in all default wrangler configs).

WAL segments (default, with the coordinator)

With the INDEX_COORDINATOR binding configured, writes are batched into segments instead of writing one R2 object per operation:

wal/{indexId}/head.json                          ← pending count, oldest op timestamp
wal/{indexId}/segments/0000000001-0000000087_chg_xxx.ndjson
wal/{indexId}/segments/open.ndjson               ← mirror of the currently open segment
...

One IndexCoordinator Durable Object per index serializes all appends: it assigns sequence numbers, accumulates operations into an in-memory open segment, and seals it to a new R2 object once the segment reaches roughly 100 operations or 256 KB. open.ndjson mirrors the not-yet-sealed segment so readers and rebuilds can see those operations without talking to the Durable Object. Head/meta mutations also go through the coordinator, so concurrent writers can never interleave sequence numbers or corrupt head.json.

Per-op entries (fallback, without the coordinator)

If the binding is absent - standalone @zbsearch/edge-core usage or the @zbsearch/edge-index-builder CLI - edge-core falls back to the original layout: one R2 object per write under wal/{indexId}/entries/{seq}_{ts}_{changeId}.ndjson, with the rebuild lock emulated by the status: 'building' flag in index metadata.

Zero-touch migration: existing per-op entries keep working - the first coordinated rebuild consumes both entries and segments. One caveat: once segments are active on an index, don't run mixed old/new writers against it (e.g. a legacy standalone edge-core deployment alongside the Worker). The binding is one-way per deployment.

Legacy deployments may still have in-place buffer segments under buffer/{indexId}/segments/; the Worker reads those during migration too.

Each WAL file contains one or more NDJSON lines (one operation per line):

{"op":"upsert","id":"sku-1","ts":"2026-07-10T12:00:00.000Z","doc":{"title":"Headphones","price":99}}
{"op":"delete","id":"sku-2","ts":"2026-07-10T12:01:00.000Z"}

The API responds immediately with 202 and a changeId. The index's pendingOps counter increments. Documents in the WAL are not searchable until the next rebuild (unless you use bulk import below).

Batch writes (POST /documents/batch) append all operations to a single WAL segment, reducing R2 PUT count. WAL reads during rebuild and search-time overlay fetch entries with bounded parallelism (10 concurrent R2 reads) instead of one object at a time.

Legacy segment rotation

Older versions grew in-place segment files (buffer/.../segments/000001.ndjson) until 64 MB, then rotated. New writes use immutable WAL objects instead - no read-modify-write on legacy buffer segments.

Read path: snapshots

Search loads a msgpack snapshot from R2:

indexes/{indexId}/{version}/snapshot.msgpack

The liveVersion field in index metadata points to the current searchable version. Snapshots are immutable - rebuild always writes a new version and atomically updates liveVersion in metadata.

Search never re-decodes the whole snapshot per request. Three layers, fastest first:

1. Per-isolate deserialized cache - an in-memory LRU holds fully loaded ZBSearch databases, keyed by the versioned snapshot key. A rebuild changes the version, so stale entries invalidate naturally. Bounded by SNAPSHOT_CACHE_MAX_ENTRIES (default 4) and SNAPSHOT_CACHE_MAX_BYTES (default ~64 MB) worker vars.
2. Workers Cache API (caches.default) - raw snapshot bytes, shared per colo. Cold isolates warm from here instead of hitting R2.
3. R2 - the durable source of truth.

Rebuild pipeline

With the coordinator binding, a rebuild performs these steps:

1. Acquire the rebuild lock on the index's IndexCoordinator Durable Object (replaces the racy status: 'building' check; a lock abandoned by a crashed rebuild can be stolen after 15 minutes)
2. Freeze the WAL: the coordinator seals the open segment and hands back all pending operations plus the list of segment keys covered by this rebuild
3. Load documents from the current live snapshot (if any)
4. Read and apply all pending WAL operations (upserts overwrite, deletes remove)
5. Build a new ZBSearch database in memory
6. Serialize to msgpack and upload to R2
7. Finalize through the coordinator: delete the frozen WAL segments, set liveVersion, documents, indexSizeBytes, status ready, update pendingOps, and release the lock

Without the binding, the same pipeline runs with per-op entries and the metadata-flag lock (steps 1, 2, and 7 use status: 'building' in meta.json instead of the Durable Object).

Before rebuild:
  liveVersion: v1  (search reads this)
  WAL: [upsert doc-A, delete doc-B, upsert doc-C]

After rebuild:
  liveVersion: v2  (new snapshot with merged state)
  WAL: (compacted - segments merged into snapshot are removed)

You can trigger rebuild manually:

curl -X POST "$WORKER_URL/v1/indexes/products/rebuild" \
  -H "authorization: Bearer $API_KEY"

Or let automation handle it (see below).

Consistency model

Typical lag depends on your rebuild strategy:

Plan for eventual consistency. If you need read-your-writes, either call rebuild after each batch of writes or poll /status until pendingOps is 0.

Rebuild strategies

1. In-Worker cron (dev / small indexes)

Configured in deploy/cloudflare/wrangler.toml:

[triggers]
crons = ["*/5 * * * *"]

[vars]
REBUILD_THRESHOLD_OPS = "500"

Every cron tick, the Worker lists indexes and rebuilds any where pendingOps >= REBUILD_THRESHOLD_OPS - unless BUILDER_WEBHOOK_URL is set.

Best for: demos, small catalogs, development.

Limitations: Worker CPU and memory limits; not suitable for multi-GB indexes. Also, every in-Worker rebuild replays the shard's entire WAL backlog in one invocation. Rolling rebuilds at the default 500-op threshold handle 100k+ documents fine, but a large one-shot backlog - bulk API ingest arriving faster than rebuilds drain, roughly 10k+ pending ops per shard - does not complete and wedges the shard at building. For bulk loads, use the external builder (zbsearch-edge-builder import) instead of the write API.

2. Manual rebuild via API

Call POST /v1/indexes/:id/rebuild from your application or a script after bulk imports.

Best for: controlled batch imports, one-off reindexing.

3. External builder CLI

Run the rebuild CLI against R2 using S3-compatible credentials:

# Rebuild one index
node packages/edge-index-builder/dist/cli.js rebuild products

# Rebuild all indexes that need it
node packages/edge-index-builder/dist/cli.js rebuild --all

# Bulk import (build snapshot locally, upload to R2)
node packages/edge-index-builder/dist/cli.js import products ./data.json --create --schema-file ./schema.json

The --all flag skips indexes that are ready with pendingOps === 0 and an existing liveVersion.

Environment variables (from deploy/cloudflare/.env):

Best for: production, large indexes, CI pipelines.

4. Webhook + GitHub Actions

Set a Worker secret so cron triggers an external runner instead of rebuilding inline:

wrangler secret put BUILDER_WEBHOOK_URL --config deploy/cloudflare/wrangler.toml
# e.g. https://api.github.com/repos/OWNER/REPO/dispatches

When pendingOps >= REBUILD_THRESHOLD_OPS, the Worker POSTs:

{ "indexId": "products", "source": "scheduler" }

Copy the workflow from deploy/github-actions/zbsearch-rebuild.yml into your repository and add secrets:

• R2_BUCKET
• R2_ACCESS_KEY_ID
• R2_SECRET_ACCESS_KEY
• R2_ENDPOINT

See Production for the full GitHub Actions setup.

Sharded indexes

For larger catalogs, a logical index can be split into N physical shards at creation time ("shards": N on POST /v1/indexes, or --shards N on the builder CLI import). Each shard <id>--shard-<i> is an ordinary index with its own WAL, snapshots, IndexCoordinator Durable Object, and rebuild thresholds - everything on this page applies per shard.

Implications for the rebuild pipeline:

• Group rebuild is serial: POST /v1/indexes/:id/rebuild on the logical index rebuilds shards one at a time - parallel shard rebuilds in one invocation exceed the 128MB isolate memory limit once shards grow past a few thousand documents (measured in production). Cron rebuilds run inline and serially for the same reason.
• Threshold scheduling is per shard: cron checks each shard's pendingOps independently, so hot shards rebuild more often than cold ones.
• Shard count is fixed: re-sharding means re-importing into a new index.
• Search fans out to ShardSearch DOs: with the SEARCH_SHARD binding, a group search runs each shard's query in that shard's own Durable Object isolate (warm snapshot cache included), so total index size is not capped by a single isolate's 128MB. Without the binding, all shards are searched in-process - fine while shards × snapshot size stays well under 128MB.

See API reference for routing, response shapes, and search caveats.

Recommended patterns

Bulk import

For large catalogs (hundreds or thousands of documents), avoid per-document API writes. Build the snapshot locally and upload directly to R2:

# Load R2 credentials from deploy/cloudflare/.env
source deploy/cloudflare/.env

# Import JSON { "data": [...] } or JSONL
node packages/edge-index-builder/dist/cli.js import products ./catalog.json \
  --create \
  --name products \
  --schema-file ./products.schema.json \
  --language english

The import command:

1. Parses documents and assigns stable IDs (slugified company_name or explicit id)
2. Builds a ZBSearch snapshot in memory
3. Uploads snapshot.msgpack to R2
4. Clears any pending WAL entries and sets liveVersion - documents are searchable immediately

For very large catalogs, shard at import time by adding --shards N (2–64): documents are bucketed deterministically by id and each shard snapshot is uploaded independently. The shard count cannot be changed later without re-importing.

For smaller batches via the API:

# 1. Batch upsert documents (single WAL file)
curl -X POST "$WORKER_URL/v1/indexes/products/documents/batch" \
  -H 'content-type: application/json' \
  -d '{"operations":[...]}'

# 2. Trigger rebuild once
curl -X POST "$WORKER_URL/v1/indexes/products/rebuild"

Streaming updates

Let writes accumulate in the WAL. Rely on cron or CI to rebuild periodically. Monitor pendingOps via /status.

Near real-time search

Lower REBUILD_THRESHOLD_OPS (e.g. 10) or rebuild after each write batch from your application. Balance rebuild cost against freshness requirements.

Storage layout reference

registry.json
indexes/{indexId}/meta.json
indexes/{indexId}/{version}/snapshot.msgpack
wal/{indexId}/head.json
wal/{indexId}/segments/*.ndjson          ← batched ops (default, coordinated)
wal/{indexId}/segments/open.ndjson       ← mirror of the open, unsealed segment
wal/{indexId}/entries/*.ndjson           ← per-op entries (fallback without DO, pre-migration)
buffer/{indexId}/head.json               ← legacy, read-only during migration
buffer/{indexId}/segments/*.ndjson       ← legacy

v1 limitations

• Full snapshots per shard - sharding (2–64 shards) splits a logical index across physical indexes, but each shard snapshot is still monolithic
• Approximate cross-shard scoring - BM25 IDF is per shard, and groupBy results are not merged on sharded indexes
• Index delete removes metadata but does not purge R2 objects automatically
• Single region R2 - snapshot locality follows your R2 bucket; Workers run globally
• Coordinator binding is one-way per deployment - once WAL segments are active on an index, don't run mixed old/new writers (e.g. a legacy standalone edge-core without the binding) against the same index

Future versions will add sharded .spb/.ivf formats and additional runtime targets (Docker, AWS).