Embedding

Embedding turns an input into a vector of numbers that captures its meaning. Inputs about the same thing get similar vectors, which powers semantic search, classification, clustering and finding duplicates.

createEmbedder runs Google’s EmbeddingGemma 2, for text in 100+ languages, images, video and audio, or all-MiniLM-L6-v2, a much smaller model for English text.

Quick start

import { createEmbedder, models, similarity } from 'runntime/zoo';

// 1. Load the model. The weights download from the Hugging Face Hub.
const embedder = await createEmbedder(models.embedding.EMBEDDINGGEMMA_2.DEFAULT);

// 2. Turn what you search into vectors.
const docs = await embedder.embedBatch(
  [
    'Two cats sleep on a pink blanket next to two TV remotes.',
    'A busy crossing with cyclists in a green bike lane.',
  ],
  { task: 'document' },
);

// 3. Turn a search into a vector and compare.
const query = await embedder.embed('cats asleep on a sofa', { task: 'query' });
docs.map((doc) => similarity(query, doc)); // [0.78, 0.57], the cats match best

// 4. Free the GPU memory when done.
embedder.dispose();

task depends on the model: EmbeddingGemma 2 uses it, all-MiniLM-L6-v2 ignores it, see Tasks. The engine must be set up first, see Getting started.

Try it yourself

Ten support answers, embedded once by the model you pick: all-MiniLM-L6-v2, a 45 MB download, or EmbeddingGemma 2 as a text-only model, 574 MB. Each search embeds the query and ranks the answers by similarity(), on your GPU.

Methods

function createEmbedder(model?: EmbedderModel, options?: LoadOptions): Promise<Embedder>;

interface Embedder {
  readonly dim: number;
  embed(input: EmbedInput, options?: EmbedOptions): Promise<Float32Array>;
  embedBatch(inputs: readonly EmbedInput[], options?: EmbedOptions): Promise<Float32Array[]>;
  dispose(): void;
}

interface EmbedOptions {
  task?: EmbeddingTask;
  dims?: number;
}

function similarity(a: Float32Array, b: Float32Array): number;

• embed(input) turns one input into one vector, see Inputs.
• embedBatch(inputs) turns many inputs into one vector each, in order. For text it is much faster than embed() in a loop.
• task and dims are per call, see Tasks and Shorter vectors.
• dim is the length of a vector without dims.
• similarity(a, b) compares two vectors: 1 for the same meaning, near 0 for unrelated. Vectors of text, images and audio compare with each other.
• dispose() frees the GPU memory.

Inputs

An input is one part, or a list of parts that makes one vector:

• text - a string.
• image - { image }, an ImageBuffer, the same as in Image classification.
• video - { video }, its frames as ImageBuffers, about one per second.
• audio - { audio }, mono samples at 16 kHz. Only the first 30 seconds count. decodeAudio reads a file, see Speech to text.

Images, video and audio need a model that reads them, see Supported models.

// Photos and text in one space.
const photos = await embedder.embedBatch([{ image: catsPhoto }, { image: footballPhoto }]);
const query = await embedder.embed('cats asleep on a sofa', { task: 'query' });
photos.map((photo) => similarity(query, photo)); // [0.76, 0.55], the cats match best

// A list makes one vector, like a product page.
const product = await embedder.embed(['Waterproof running shoes.', { image: shoePhoto }]);

Text longer than maxTokens is cut. An input with images, video or audio that does not fit is an error.

Tasks

task puts the instruction the model was trained with before the text. A search embeds its queries with query and what it searches with document. Embed images, video and audio without a task.

Task	For
query	search queries, compared with document
questionAnswering	questions, compared with document
factChecking	claims, compared with document
codeRetrieval	descriptions of code, compared with document code
document	what a search searches
classification	every text being classified
clustering	every text being clustered
similarity	every text being compared

Without task a text gets no instruction, which works but matches less well.

Shorter vectors

dims cuts a vector to fewer numbers, for less storage at a small cost in quality. Compare only vectors of the same length. The lengths a model allows are in Supported models, and the cost for EmbeddingGemma 2 is in its scores per length.

// A third of the storage.
const short = await embedder.embed('cats asleep on a sofa', { dims: 256 });
short.length; // 256

Options

The first argument is the model config, a plain object with the fields below. models holds ready-made ones: DEFAULT, the f16 weights run in f16 where the device has shader-f16 and in f32 elsewhere; F16, the same files with f16 forced; F32, the f32 weights run in f32. Pass one as is, copy it with a field changed, or point at your own files.

interface EmbedderModel {
  readonly modelPath?: ModelPath;
  readonly visionPath?: ModelPath | null;
  readonly audioPath?: ModelPath | null;
  readonly tokenizerPath?: string;
  readonly dtype?: 'f32' | 'f16';
  readonly maxTokens?: number;
  readonly imageTokens?: 70 | 140 | 280 | 560 | 1120;
}

type ModelPath = string | RangeSource; // a URL, or bytes you already have

Examples:

import { createEmbedder, models } from 'runntime/zoo';

// The small model, for English text.
await createEmbedder(models.embedding.ALL_MINILM_L6_V2.DEFAULT);

// EmbeddingGemma 2 for text only: a smaller download.
await createEmbedder({
  ...models.embedding.EMBEDDINGGEMMA_2.DEFAULT,
  visionPath: null,
  audioPath: null,
});

• modelPath - the weights, a model.safetensors file. The model is read from the file.
• visionPath - the weights for images and video. null skips the download, and images and video become an error.
• audioPath - the same for audio.
• tokenizerPath - the tokenizer.json file.
• dtype - 'f16' or 'f32'. f16 runs the model in half precision, half the GPU memory, and needs a device with shader-f16. Default: f16 where the device has it, f32 elsewhere. Either loads any weight file.
• maxTokens - longest input in tokens. The default and the most depend on the model, see Supported models.
• imageTokens - most tokens one image takes, 280 by default. More tokens keep more detail and take longer.

Every field is optional. A missing one comes from the model’s DEFAULT entry. Without modelPath, the model is EmbeddingGemma 2.

Loading options

The second argument controls the download. Every field is optional.

interface LoadOptions {
  cache?: WeightCache;
  cacheId?: string;
  onProgress?: (name: string, doneBytes: number, totalBytes: number) => void;
  onBytes?: (chunkBytes: number) => void;
  signal?: AbortSignal;
}

Example:

import { createEmbedder, createOpfsCache, models } from 'runntime/zoo';

const embedder = await createEmbedder(models.embedding.EMBEDDINGGEMMA_2.DEFAULT, {
  cache: await createOpfsCache('my-app'),
  onProgress: (name, done, total) => console.log(`${Math.round((100 * done) / total)}%`),
});

• cache - keeps the downloaded weights and tokenizer in the browser, so the next visit loads them without the network. createOpfsCache(name) is the built-in one.
• onProgress - called while the weights upload to the GPU, with the tensor name, bytes done and bytes total. Use it for a progress bar.
• signal - an AbortSignal. Abort it to stop the load between steps. A step already running, like the download or the GPU upload, finishes first.

Supported models

Model	Config	Inputs	Tasks	Dimensions	Max tokens	Weights
EmbeddingGemma 2	models.embedding.EMBEDDINGGEMMA_2	text in 100+ languages, images, video, audio	yes	768, 512, 256, 128	2048	f16 1.5 GB, f32 2.7 GB
all-MiniLM-L6-v2	models.embedding.ALL_MINILM_L6_V2	English text	no	384	256, at most 512	f16 45 MB, f32 91 MB