Insert Data

Whenever we create a database with ZBSearch, we must specify a schema, which represents the entry we are going to index.

Let's say our database and schema look like this:

import { create, insert } from "zbsearch";

const movieDB = create({
  schema: {
    title: "string",
    director: "string",
    plot: "string",
    year: "number",
    isFavorite: "boolean",
  },
});

(Read more about database creation on the create page)

Insert

Data insertion in ZBSearch is quick and intuitive:

const thePrestigeId = insert(movieDB, {
  title: "The prestige",
  director: "Christopher Nolan",
  plot: "Two friends and fellow magicians become bitter enemies after a sudden tragedy. As they devote themselves to this rivalry, they make sacrifices that bring them fame but with terrible consequences.",
  year: 2006,
  isFavorite: true,
});

const bigFishId = insert(movieDB, {
  title: "Big Fish",
  director: "Tim Burton",
  plot: "Will Bloom returns home to care for his dying father, who had a penchant for telling unbelievable stories. After he passes away, Will tries to find out if his tales were really true.",
  year: 2004,
  isFavorite: true,
});

const harryPotterId = insert(movieDB, {
  title: "Harry Potter and the Philosopher's Stone",
  director: "Chris Columbus",
  plot: "Harry Potter, an eleven-year-old orphan, discovers that he is a wizard and is invited to study at Hogwarts. Even as he escapes a dreary life and enters a world of magic, he finds trouble awaiting him.",
  year: 2001,
  isFavorite: false,
});

If you have a lot of records, we suggest using the insertMultiple function as following:

const docs = [
  {
    title: "The prestige",
    director: "Christopher Nolan",
    plot: "Two friends and fellow magicians become bitter enemies after a sudden tragedy. As they devote themselves to this rivalry, they make sacrifices that bring them fame but with terrible consequences.",
    year: 2006,
    isFavorite: true,
  },
  {
    title: "Big Fish",
    director: "Tim Burton",
    plot: "Will Bloom returns home to care for his dying father, who had a penchant for telling unbelievable stories. After he passes away, Will tries to find out if his tales were really true.",
    year: 2004,
    isFavorite: true,
  },
  {
    title: "Harry Potter and the Philosopher's Stone",
    director: "Chris Columbus",
    plot: "Harry Potter, an eleven-year-old orphan, discovers that he is a wizard and is invited to study at Hogwarts. Even as he escapes a dreary life and enters a world of magic, he finds trouble awaiting him.",
    year: 2001,
    isFavorite: false,
  },
];

insertMultiple(movieDB, docs);

Inserting a large number of documents in a loop could potentially block the event loop. Instead insertMultiple handles this case better.

You can pass a third, optional, parameter to change the batch size (default: 1000). We recommend keeping this number as low as possible to avoid blocking the event loop. The batchSize refers to the maximum number of insert operations to perform before yielding the event loop.

insertMultiple(movieDB, docs, 500);

Non-blocking insertion

insertMultiple is synchronous: it returns only once every document has been indexed. Indexing tens of thousands of documents up front therefore freezes the page for as long as the build takes.

insertMultipleAsync does the same work in batches and hands control back to the event loop between them, so rendering, input handling and timers keep getting a turn:

import { insertMultipleAsync } from "zbsearch";

const ids = await insertMultipleAsync(movieDB, docs);

It accepts an options object:

const ids = await insertMultipleAsync(movieDB, docs, {
  batchSize: 100, // documents to index before yielding, default: 100
  language: "english", // as in `insertMultiple`
  skipHooks: false, // as in `insertMultiple`
  onProgress: ({ processed, total }) => {
    progressBar.value = processed / total;
  },
});

The default batchSize is 100, lower than insertMultiple's 1000: this variant optimizes for how long the main thread stays blocked at a stretch rather than for raw throughput. Indexing 30,000 documents on Node 24 (npm run benchmark:responsiveness in the repo) shows what the knob buys you:

Variant	Total	Longest block
`insertMultiple` (sync)	954ms	the entire run - nothing else could run
`insertMultipleAsync`, `batchSize: 1000`	733ms	52.2ms
`insertMultipleAsync`, `batchSize: 100` (default)	737ms	7.0ms
`insertMultipleAsync`, `batchSize: 25`	814ms	5.3ms

The total work is unchanged - what collapses is the longest uninterrupted stretch, from the whole build down to a few milliseconds. Note that batchSize: 1000 still crosses the 50ms mark Chrome reports as a long task, which is why the default is 100: it stays comfortably inside a 60fps frame budget at negligible cost. Lower it further when individual documents are large.

The resolved ids are in the same order as the documents you passed, exactly as with insertMultiple. If a document fails to insert the promise rejects, and the documents indexed before it stay in the database.

The indexing work still runs on the calling thread, so the total build time is unchanged - what changes is that the main thread is never blocked for more than one batch at a time. To keep indexing off the main thread entirely, run ZBSearch inside a Web Worker.

Validation rules

Defining the schema at database creation time, ZBSearch validates all the inserted documents following those rules:

throw an error if a field has an unexpected type
allow missing fields or fields set to undefined
allow extra fields ignoring them

So the following document will be accepted:

import { create, insert } from "zbsearch";

const movieDB = create({
  schema: {
    title: "string",
    year: "number",
  },
});

insert(movieDB, {
  title: "The prestige",
  // `year` field is missing but it's ok
  // year: 2006,
  // Extra fields `director` and `isFavorite` will not be indexed
  director: "Christopher Nolan",
  isFavorite: true,
});

Custom document IDs

If the id field is not found, ZBSearch will generate a random id for the document. To provide a custom ID for a document, see the components page.

ZBSearch automatically uses the id field of the document, if found.

That means that given the following document and schema:

import { create, search } from "zbsearch";

const db = create({
  schema: {
    id: "string",
    author: "string",
    quote: "string",
  },
});

insert(db, {
  id: "73cbcc79-2203-49b8-bb52-60d8e9a66c5f",
  author: "Fernando Pessoa",
  quote: "I wasn't meant for reality, but life came and found me",
});

the document will be indexed with the following id: 73cbcc79-2203-49b8-bb52-60d8e9a66c5f.

Beware
If you try to insert two documents with the same ID, ZBSearch will throw an error.

Remote document storing

By default ZBSearch keeps a copy of the inserted document in memory (and in the serialized data) to speed up search performance.

If this is not acceptable, you can provide a custom documentsStore component which will be responsible to store and fetch documents from another location (local or remote).

The code example below is an example that implements a proxy: when a document is requested, the code finds it on a location of the filesystem. We assume each document has an id field which disable ZBSearch random ID generation.

You can replace the file related operations with your custom code.

import { readFile, readdir, writeFile, rm } from "node:fs/promises";
import { resolve } from "node:path";
import { create } from "zbsearch";

const ROOT_LOCATION = "/var/db/orama-example";

async function getDocument(id) {
  return JSON.parse(
    await readFile(resolve(ROOT_LOCATION, `${id}.json`), "utf-8")
  );
}

async function listDocuments() {
  const allFiles = await readdir(ROOT_LOCATION);

  return allFiles.filter((id) => id.endsWith(".json"));
}

const database = create({
  schema: {
    title: "string",
    director: "string",
  },
  components: {
    // override partially the default documents store
    documentsStore: {
      create() {
        return {};
      },
      load(raw) {
        return {};
      },
      save(store) {
        return {};
      },
      get(_, id) {
        return getDocument(id);
      },
      getMultiple(_, ids) {
        return Promise.all(
          ids.map(async (id) => {
            return JSON.parse(await getDocument(id));
          })
        );
      },
      async getAll() {
        const docs = await listDocuments();

        return Promise.all(
          docs.map(async (id) => {
            return JSON.parse(await getDocument(id));
          })
        );
      },
      store() {
        // No-op
      },
      remove() {
        // No-op
      },
      async count() {
        const docs = await listDocuments();

        return docs.count;
      },
    },
  },
});

Insert

Non-blocking insertion

Validation rules

Custom document IDs

Remote document storing

On this page