Getting started

ruNNtime zoo is a set of ready-to-run AI models for the web: text embeddings, speech to text, computer vision and more. They run completely offline, using own AI kernels, written in TypeGPU. Because of that, zoo works in any environment with WebGPU: the web, React Native, Electron and more. The kernels are plain TypeGPU code, so there is no custom WASM binary to download at runtime.

What are the benefits?

Private - the data never leaves the device. You don’t have to host a server, and you don’t need any API keys.
Fast - ruNNtime is faster than existing alternatives. See Benchmarks for comparison with other solutions.
Simple - one function loads a model, one method runs it.
Offline after the first load - the weights download once and can be cached.

What is ruNNtime?

ruNNtime is a framework for deploying AI models with TypeGPU. It has two packages:

zoo - the documentation you are reading. Ready-to-use model implementations across the AI stack: NLP, speech, vision.
core - This part is not yet officially released to the public (stay tuned 👀). It is the engine powering zoo. Bare AI primitives, so a model can be composed from numerical blocks and ported straight from a research framework.

Installation

1. Install TypeGPU

ruNNtime is built on TypeGPU: the underlying kernels are TypeGPU functions, so you only need TypeGPU itself installed. No bundler plugin is required to use the library.

Follow the TypeGPU installation guide.

2. Install ruNNtime

npm install runntime

pnpm add runntime

yarn add runntime

Requirements

Web - a browser with WebGPU: Chrome, Edge, Safari 26 and newer, recent Firefox. See caniuse for the details.
Electron - ships Chromium, so the web setup works as is.
React Native - through react-native-webgpu. Call its installWebGPU() first, then request the device yourself and wrap it with tgpu.initFromDevice({ device }) before passing it to initRunntime().

Set up the engine

Once per page, before you create any model:

import { 
function initRunntime(root: TgpuRoot): RunntimeExecutor
initRunntime } from 'runntime/zoo';
import 
const tgpu: {
    const: {
        <TDataType extends AnyData>(dataType: TDataType, value: InferGPU<TDataType>): TgpuConst<TDataType>;
        <TElement extends AnyWgslData>(dataType: (elementCount: number) => WgslArray<TElement>, value: InferGPU<WgslArray<TElement>>): TgpuConst<WgslArray<TElement>>;
    };
    fn: {
        <Args extends BaseData[] | []>(argTypes: Args, returnType?: undefined): TgpuFnShell<Args, Void>;
        <Args extends BaseData[] | [], Return extends BaseData>(argTypes: Args, returnType: Return): TgpuFnShell<Args, Return>;
        <T extends AnyFn>(inner: T): TgpuGenericFn<T>;
    };
    ... 17 more ...;
    '~unstable': typeof import("/home/runner/work/runntime/runntime/node_modules/.pnpm/typegpu@0.12.5/node_modules/typegpu/tgpuUnstable");
}
tgpu from 'typegpu';

function initRunntime(root: TgpuRoot): RunntimeExecutor
initRunntime(await 
const tgpu: {
    const: {
        <TDataType extends AnyData>(dataType: TDataType, value: InferGPU<TDataType>): TgpuConst<TDataType>;
        <TElement extends AnyWgslData>(dataType: (elementCount: number) => WgslArray<TElement>, value: InferGPU<WgslArray<TElement>>): TgpuConst<WgslArray<TElement>>;
    };
    fn: {
        <Args extends BaseData[] | []>(argTypes: Args, returnType?: undefined): TgpuFnShell<Args, Void>;
        <Args extends BaseData[] | [], Return extends BaseData>(argTypes: Args, returnType: Return): TgpuFnShell<Args, Return>;
        <T extends AnyFn>(inner: T): TgpuGenericFn<T>;
    };
    ... 17 more ...;
    '~unstable': typeof import("/home/runner/work/runntime/runntime/node_modules/.pnpm/typegpu@0.12.5/node_modules/typegpu/tgpuUnstable");
}
tgpu.
init: (options?: InitOptions) => Promise<TgpuRoot>
init({ 
device?: (GPUDeviceDescriptor & {
    optionalFeatures?: Iterable<GPUFeatureName>;
}) | undefined
device: { 
optionalFeatures?: Iterable<GPUFeatureName> | undefined
optionalFeatures: ['subgroups', 'shader-f16'] } }));

initRunntime() connects the engine to a WebGPU device. Both features are requested as optional, so the call succeeds on devices without them. Many kernels run much faster with subgroups. Every vision model - object detection, instance segmentation, pose, depth estimation and image classification - runs in half precision and needs shader-f16. Without it the create call throws UNSUPPORTED_DEVICE, see Error handling. Embedding, speech to text and the privacy filter run in f32 on such devices. If your app already uses TypeGPU, pass the root you have.

Pick a model

Every model has its own factory function, like createObjectDetector(...), and its own methods. The model’s page shows how to load it, call it and free it:

Embedding - text, images, video and audio to vectors in one space, for search and similarity.
Speech to text - recorded speech to text.
Privacy filter - finds personal data in text.
Object detection - finds objects in an image.
Instance segmentation - finds objects in an image, each with a pixel mask.
Pose & keypoints - finds people and their body landmarks in an image.
Depth estimation - how far each pixel of an image is.
Image classification - what an image shows.

Every factory and every model method throws a RunntimeError with a code when something goes wrong. See Error handling for the codes and how to catch them.

Coming from transformers.js?

Already using transformers.js and don’t want to move your whole codebase to a new framework? Keep your pipeline() code and swap the engine underneath. By default transformers.js runs on ONNX Runtime, which pays for its generality in speed. One initRunntimeBackend() call replaces that backend with ruNNtime. It only changes anything for the models ruNNtime supports, everything else keeps loading on ONNX Runtime.

import { 
function initRunntimeBackend(opts?: RegisterRunntimeBackendOpts): Promise<TgpuRoot>
initRunntimeBackend } from 'runntime/zoo/transformers';
import { 
function pipeline<T extends PipelineType>(task: T, model?: string, { progress_callback, config, cache_dir, local_files_only, revision, device, dtype, subfolder, use_external_data_format, model_file_name, session_options, }?: PretrainedModelOptions): Promise<AllTasks[T]>
pipeline } from '@huggingface/transformers';

await 
function initRunntimeBackend(opts?: RegisterRunntimeBackendOpts): Promise<TgpuRoot>
initRunntimeBackend({ 
RegisterRunntimeBackendOpts.fallbackToOnnx?: boolean | undefined
fallbackToOnnx: true });

const 
const pipe: FeatureExtractionPipeline
pipe = await 
pipeline<"feature-extraction">(task: "feature-extraction", model?: string, { progress_callback, config, cache_dir, local_files_only, revision, device, dtype, subfolder, use_external_data_format, model_file_name, session_options, }?: PretrainedModelOptions): Promise<FeatureExtractionPipeline>
pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');

See migrating from transformers.js for the supported models and the fallback rules.