Depth estimation

Depth estimation tells how far away each part of an image is. You pass in the pixels and get back a depth map: one number per pixel, smaller for what is nearer to the camera. The numbers are relative, not meters: they order the scene from near to far within one image.

createDepthEstimator runs DepthART on the GPU.

Through the transformers.js plugin, DepthART runs under the Depth Anything V2 model ids, onnx-community/depth-anything-v2-small and -base. transformers.js has no DepthART, so a depth-estimation pipeline with one of those ids runs DepthART, not Depth Anything. The map comes back at 448×448 with smaller for nearer, the opposite of Depth Anything.

Quick start

import { createDepthEstimator, imageBufferFromImageData, models } from 'runntime/zoo';

// 1. Load the model once. The weights download from the Hugging Face Hub.
const estimator = await createDepthEstimator(models.depthEstimation.DEPTHART.DEFAULT);

// 2. Get the pixels of an image, here from a canvas the picture is drawn on.
const ctx = canvas.getContext('2d')!;
const image = imageBufferFromImageData(ctx.getImageData(0, 0, canvas.width, canvas.height));

// 3. Estimate the depth.
const depth = await estimator.estimateDepth(image);

// 4. Use it: one value per pixel of the 448×448 map, row by row, smaller = nearer.
const { width, height, data } = depth;
const center = data[(height / 2) * width + width / 2];

// 5. Free the GPU memory when done.
estimator.dispose();

The example reads one picture off a canvas. A video or a camera works the same way: draw each frame on the canvas and call estimateDepth again, the estimator is loaded once.

The engine must be set up first, see Getting started. The model runs in half precision, so the device needs shader-f16.

Try it yourself

Your camera, frame by frame through encodeEstimateDepth() on your GPU, as in GPU in, GPU out: the frame never leaves the GPU and the map is drawn straight into the canvas, nearest black, farthest white. No camera? The sample picture goes through the same call.

Results

estimateDepth returns one depth map:

interface DepthMap {
  width: number; // 448
  height: number; // 448
  data: Float32Array; // width × height values, row by row, smaller = nearer
}

The map covers the whole image, scaled to the model’s 448×448 input, so the value for a pixel (x, y) of the image sits at data[Math.floor(y * height / image.height) * width + Math.floor(x * width / image.width)].

Methods

function createDepthEstimator(
  model?: DepthEstimatorModel,
  options?: LoadOptions,
): Promise<DepthEstimator>;

interface DepthEstimator {
  readonly inputSize: number;
  readonly schemas: { readonly input: DepthInput; readonly output: DepthMapData };
  estimateDepth(image: ImageBuffer): Promise<DepthMap>;
  encodeEstimateDepth(options: EncodeEstimateDepthOptions): void;
  dispose(): void;
}

• estimateDepth(image) returns one depth map for the image. image is an ImageBuffer of any size. Build one with imageBufferFromImageData(imageData) from a canvas, or imageBuffer(bytes, width, height, format?) from raw rgba, rgb, bgra, bgr or gray bytes.
• encodeEstimateDepth(options) reads an image already on the GPU and writes the map to the GPU, see GPU in, GPU out.
• inputSize is the width and height of every returned map, 448.
• schemas are the TypeGPU layouts of the model input and the depth map, for buffers passed to encodeEstimateDepth.
• dispose() frees the GPU memory.

GPU in, GPU out

When the image is already on the GPU, a video frame or the output of your own shader, encodeEstimateDepth skips the CPU: no pixels are read back, and the map goes straight into a texture or a buffer you own.

import { createDepthEstimator, initRunntime, models } from 'runntime/zoo';
import tgpu from 'typegpu';

// 1. Set up the engine and load the model once.
const root = await tgpu.init({ device: { optionalFeatures: ['subgroups', 'shader-f16'] } });
initRunntime(root);
const estimator = await createDepthEstimator(models.depthEstimation.DEPTHART.DEFAULT);

// 2. Draw into a canvas on the same device.
const context = canvas.getContext('webgpu')!;
context.configure({ device: root.device, format: navigator.gpu.getPreferredCanvasFormat() });

// 3. Each frame: video in, depth in the canvas, nearest black, farthest white.
function frame() {
  estimator.encodeEstimateDepth({
    source: root.device.importExternalTexture({ source: video }),
    target: context.getCurrentTexture(),
    normalize: 'minmax',
  });
  video.requestVideoFrameCallback(frame);
}
video.requestVideoFrameCallback(frame);

The call records the work and submits it before it returns, so it does not wait for the GPU. Work you submit after it sees the map.

Source - one of:

• a texture: a GPUTexture, a TypeGPU texture with sampled usage, or a GPUExternalTexture from a video. It is fitted like an ImageBuffer, with the estimator’s resizeMode, and each value 0-1 counts as a byte 0-255. Formats: rgba8unorm, bgra8unorm, rgb10a2unorm, rgba16float, rgba32float. sRGB formats are rejected: the GPU would turn their values linear, which the model was not trained on.
• a buffer holding the model input as is: 3×448×448 f32, normalized RGB, one plane per channel. A raw GPUBuffer needs COPY_SRC usage.

Target - one of:

• a texture: a GPUTexture or a TypeGPU texture with render usage, of any size and any float-renderable format. The map is stretched over it, the same value in every color channel. normalize: 'minmax' maps each map to 0-1, nearest = 0. The default 'none' writes the raw values, which an unorm format clamps to 0-1.
• a buffer: the 448×448 f32 map, row by row, the same values as DepthMap.data. A raw GPUBuffer needs COPY_DST usage.

With TypeGPU, schemas gives the buffer its type and size:

// [row][column] f32, 448 × 448.
const depth = root.createBuffer(estimator.schemas.output).$usage('storage');

const encoder = root['~unstable'].createCommandEncoder();
estimator.encodeEstimateDepth({ source: cameraTexture, target: depth, encoder });
// ...record your own passes that read `depth` here...
encoder.submit();

encoder - optional. Pass a GPUCommandEncoder or a TypeGPU command encoder to record the estimate together with your own passes, or several estimates, into one command buffer. You then finish and submit it yourself, before dispose().

Every resource must come from the device passed to initRunntime(). A wrong format, usage or size throws INVALID_ARGUMENT.

Options

The first argument is the model config, a plain object with the fields below. models.depthEstimation.DEPTHART holds ready-made ones: S (13 MB, the DEFAULT) and B (24 MB, more detail, slower). Pass one as is, copy it with a field changed, or point at your own safetensors export of a DepthART relative checkpoint.

interface DepthEstimatorModel {
  readonly modelPath?: ModelPath;
  readonly variant?: 'b' | 's';
  readonly resizeMode?: 'stretch' | 'letterbox' | 'crop'; // default 'stretch'
}

type ModelPath = string | RangeSource; // a URL, or bytes you already have

Examples:

import { createDepthEstimator, models } from 'runntime/zoo';

// The base size.
await createDepthEstimator(models.depthEstimation.DEPTHART.B);

// Your own copy of the file.
await createDepthEstimator({ modelPath: '/models/depthart-s/model.safetensors' });

• modelPath - the weights, a model.safetensors file. Default: models.depthEstimation.DEPTHART.DEFAULT.modelPath.
• variant - the model size, b or s. Read from the weights by default.
• resizeMode - how the image fits the model input. stretch (default) scales width and height separately, so the map covers the whole image. letterbox pads, crop cuts the centered square.

Loading options

The second argument controls the download. Every field is optional.

interface LoadOptions {
  cache?: WeightCache;
  cacheId?: string;
  onProgress?: (name: string, doneBytes: number, totalBytes: number) => void;
  onBytes?: (chunkBytes: number) => void;
  signal?: AbortSignal;
}

Example:

import { createDepthEstimator, createOpfsCache, models } from 'runntime/zoo';

const estimator = await createDepthEstimator(models.depthEstimation.DEPTHART.DEFAULT, {
  cache: await createOpfsCache('my-app'),
  onProgress: (name, done, total) => console.log(`${Math.round((100 * done) / total)}%`),
});

• cache - keeps the downloaded weights in the browser, so the next visit loads them without the network. createOpfsCache(name) is the built-in one.
• onProgress - called while the weights upload to the GPU, with the tensor name, bytes done and bytes total. Use it for a progress bar.
• signal - an AbortSignal. Abort it to stop the load between steps. A step already running, like the download or the GPU upload, finishes first.

Supported models

Model	Config	Weights	Input	Output
DepthART s	models.depthEstimation.DEPTHART.S	f16, 12.6 MB	448×448	448×448 relative depth
DepthART b	models.depthEstimation.DEPTHART.B	f16, 23.6 MB	448×448	448×448 relative depth