Depth Estimation

Depth estimation predicts a depth value for every pixel in an image. The pipeline returns a floating-point depth map matching the original image dimensions, which can be used for portrait blur, photo animation, and scene understanding.

Quick Start

The useDepthEstimator hook manages model downloading, initialization, and lifecycle:

import { models, useDepthEstimator } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function MyComponent() {
  const estimator = useDepthEstimator(models.depthEstimation.DEPTH_ANYTHING_V2_SMALL.DEFAULT);

  // Hook state:
  // estimator.isReady          — true once model is downloaded and loaded in memory
  // estimator.downloadProgress — 0 to 100 download progress
  // estimator.error            — Error instance if download or load failed
  // estimator.resource         — resolved config with all URLs replaced by local file paths

  const handleEstimate = async (imageBuffer: ImageBuffer) => {
    if (!estimator.isReady || !estimator.estimateDepth) return;

    // Run inference on a background thread
    const depth = await estimator.estimateDepth(imageBuffer);
    const x = Math.floor(depth.width / 2);
    const y = Math.floor(depth.height / 2);
    console.log('Center depth:', depth.data[y * depth.width + x]);
  };

  // Trigger handleEstimate from an image picker, button press, or camera frame
}

Full Interactive Example

See apps/computer-vision/app/depthEstimation/index.tsx for a complete screen with model selection, photo capture, depth visualization, interactive photo effects, and latency tracking.

Output Format

estimateDepth() returns a DepthMap object:

type DepthMap = {
  /** Raw depth values in row-major order: data[y * width + x] */
  readonly data: Float32Array;
  /** Width of the depth map in pixels */
  readonly width: number;
  /** Height of the depth map in pixels */
  readonly height: number;
};

The map contains one float per source image pixel. Each result owns its buffer and remains valid after subsequent inference or pipeline disposal.

Values retain the model's original scale. For visualization, normalize a copy of the data and apply a color map in your application.

Configuration & Options

The DepthEstimatorOptions in modelOpts configure input preprocessing and output resizing. Hosted presets include these settings:

Option	Description
resizeMode	Resize the full image to the model input dimensions. Currently only 'stretch' is supported.
interpolation	Interpolation used to resize the input image.
normalizeOpts	Input channel scaling and normalization.
outInterpolation	Interpolation used to resize the float depth map to the source image dimensions.

ZipDepth applies ImageNet normalization inside the model, so its preset only scales input pixel values to [0, 1].

There are no per-inference options. To change preprocessing for a different model, supply a custom model configuration.

Imperative API

For background processing or manual lifecycle management outside React components, create the estimator using createDepthEstimator:

import { createDepthEstimator, download, models } from 'react-native-executorch';

// Download and cache model assets before creating the pipeline
const model = await download(models.depthEstimation.DEPTH_ANYTHING_V2_SMALL.DEFAULT);
const estimator = await createDepthEstimator(model);

try {
  const depth = await estimator.estimateDepth(imageBuffer);
  console.log('Depth map dimensions:', depth.width, depth.height);
} finally {
  // Always release native resources when finished
  estimator.dispose();
}

Synchronous Execution

For worklet-based processing, createDepthEstimator exposes a synchronous estimateDepthWorklet function:

// Called synchronously on the worklet thread
const depth = estimator.estimateDepthWorklet(frameBuffer);

See Worklets & Threading for details on worklet execution contexts.

Available Models

Ready-to-use models are available in models.depthEstimation:

Model Family	Variants	Size Range	Supported Backends	Notes
Depth Anything V2 Small	See	35.5 MB – 99.0 MB	XNNPACK (CPU), Core ML (Apple), Vulkan (Android)	ViT-S/14 backbone with a 518 × 518 input and relative inverse depth output.
Depth Anything 3 Small	See	40.7 MB – 106.5 MB	XNNPACK (CPU), Core ML (Apple), Vulkan (Android)	Single-image relative depth with a 518 × 518 input.
ZipDepth Base	See	12.8 MB – 24.9 MB	XNNPACK (CPU), Core ML (Apple), Vulkan (Android)	Fast preset with a 384 × 384 input and relative inverse depth output.

Depth Anything V2 and ZipDepth return relative inverse depth (larger means nearer), while Depth Anything 3 returns relative depth (larger means farther). All presets return raw float values through the same DepthMap API.

Using Custom Models

Pass a DepthEstimatorModel configuration to useDepthEstimator or download it before calling createDepthEstimator:

import { createDepthEstimator, download, IMAGENET_NORM } from 'react-native-executorch';

const model = await download({
  modelPath: 'https://example.com/my-depth-model.pte',
  modelOpts: {
    resizeMode: 'stretch' as const,
    interpolation: 'cubic' as const,
    normalizeOpts: IMAGENET_NORM,
    outInterpolation: 'linear' as const,
  },
});
const estimator = await createDepthEstimator(model);

Choose normalization and interpolation settings for your model. The pipeline accepts static float32 RGB inputs [1, 3, H, W] with outputs [1, OH, OW], or unbatched inputs [3, H, W] with outputs [OH, OW]. Input and output spatial dimensions may differ. Output values retain the custom model's meaning and units.

The pipeline verifies the exported input and output shapes automatically. See Exporting Custom Models for export requirements.

API Reference

Hooks & Pipelines

• useDepthEstimator() — React hook for model downloading, state, and lifecycle.
• createDepthEstimator() — Imperative factory for depth estimation pipelines.

Types & Options

• DepthEstimator — Runner interface with estimateDepth and estimateDepthWorklet.
• DepthMap — Floating-point depth data with width and height.
• DepthEstimatorModel — Model configuration spec.
• DepthEstimatorOptions — Input preprocessing and output interpolation options.
• ImageBuffer — Input image buffer structure.

Model Presets & Constants

• models.depthEstimation — Pre-configured depth estimation models registry.
• IMAGENET_NORM — Input normalization used by Depth Anything V2 and V3 presets.

Source Code

View the implementation on GitHub:

• src/extensions/cv/tasks/depthEstimation.ts ↗