Version: 0.8.x

Class: ObjectDetectionModule<T>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:86

Generic object detection module with type-safe label maps.

Extends

VisionLabeledModule<Detection<ResolveLabels<T>>[], ResolveLabels<T>>

Type Parameters

T

T extends ObjectDetectionModelName | LabelEnum

Either a built-in model name (e.g. 'ssdlite-320-mobilenet-v3-large') or a custom LabelEnum label map.

Properties

generateFromFrame()

generateFromFrame: (frameData, ...args) => any

Defined in: modules/BaseModule.ts:53

Process a camera frame directly for real-time inference.

This method is bound to a native JSI function after calling load(), making it worklet-compatible and safe to call from VisionCamera's frame processor thread.

Performance characteristics:

Zero-copy path: When using frame.getNativeBuffer() from VisionCamera v5, frame data is accessed directly without copying (fastest, recommended).
Copy path: When using frame.toArrayBuffer(), pixel data is copied from native to JS, then accessed from native code (slower, fallback).

Usage with VisionCamera:

const frameOutput = useFrameOutput({
  pixelFormat: 'rgb',
  onFrame(frame) {
    'worklet';
    // Zero-copy approach (recommended)
    const nativeBuffer = frame.getNativeBuffer();
    const result = model.generateFromFrame(
      { nativeBuffer: nativeBuffer.pointer, width: frame.width, height: frame.height },
      ...args
    );
    nativeBuffer.release();
    frame.dispose();
  }
});

Parameters

frameData

Frame

Frame data object with either nativeBuffer (zero-copy) or data (ArrayBuffer)

args

...any[]

Additional model-specific arguments (e.g., threshold, options)

Returns

any

Model-specific output (e.g., detections, classifications, embeddings)

See

Frame for frame data format details

Inherited from

VisionLabeledModule.generateFromFrame

labelMap

protected readonly labelMap: ResolveLabels

Defined in: modules/computer_vision/VisionLabeledModule.ts:42

Inherited from

VisionLabeledModule.labelMap

nativeModule

nativeModule: any = null

Defined in: modules/BaseModule.ts:16

Internal

Native module instance (JSI Host Object)

Inherited from

VisionLabeledModule.nativeModule

Accessors

runOnFrame

Get Signature

get runOnFrame(): (frame, isFrontCamera, options?) => Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]

Defined in: modules/computer_vision/ObjectDetectionModule.ts:155

Override runOnFrame to provide an options-based API for VisionCamera integration.

Throws

If the underlying native worklet is unavailable (should not occur on a loaded module).

Returns

A worklet function for frame processing.

(frame, isFrontCamera, options?): Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]

Parameters

frame

Frame

isFrontCamera

boolean

options?

ObjectDetectionOptions<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>

Returns

Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]

Overrides

VisionLabeledModule.runOnFrame

Methods

delete()

delete(): void

Defined in: modules/BaseModule.ts:81

Unloads the model from memory and releases native resources.

Always call this method when you're done with a model to prevent memory leaks.

Returns

void

Inherited from

VisionLabeledModule.delete

forward()

forward(input, options?): Promise<Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:247

Executes the model's forward pass to detect objects within the provided image.

Supports two input types:

String path/URI: File path, URL, or Base64-encoded string
PixelData: Raw pixel data from image libraries (e.g., NitroImage)

Parameters

input

A string image source (file path, URI, or Base64) or a PixelData object.

string | PixelData

options?

Optional configuration for detection inference. Includes detectionThreshold, inputSize, and classesOfInterest.

Returns

Promise<Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]>

A Promise resolving to an array of Detection objects.

Throws

If the model is not loaded or if an invalid inputSize is provided.

Example

const detections = await model.forward('path/to/image.jpg', {
  detectionThreshold: 0.7,
  inputSize: 640,  // For YOLO models
  classesOfInterest: ['PERSON', 'CAR'],
});

Overrides

VisionLabeledModule.forward

forwardET()

protected forwardET(inputTensor): Promise<TensorPtr[]>

Defined in: modules/BaseModule.ts:62

Internal

Runs the model's forward method with the given input tensors. It returns the output tensors that mimic the structure of output from ExecuTorch.

Parameters

inputTensor

TensorPtr[]

Array of input tensors.

Returns

Promise<TensorPtr[]>

Array of output tensors.

Inherited from

VisionLabeledModule.forwardET

getAvailableInputSizes()

getAvailableInputSizes(): readonly number[] | undefined

Defined in: modules/computer_vision/ObjectDetectionModule.ts:146

Returns the available input sizes for this model, or undefined if the model accepts any size.

Returns

readonly number[] | undefined

An array of available input sizes, or undefined if not constrained.

Example

const sizes = model.getAvailableInputSizes(); // [384, 512, 640] for YOLO models, or undefined for RF-DETR

getInputShape()

getInputShape(methodName, index): Promise<number[]>

Defined in: modules/BaseModule.ts:72

Gets the input shape for a given method and index.

Parameters

methodName

string

method name

index

number

index of the argument which shape is requested

Returns

Promise<number[]>

The input shape as an array of numbers.

Inherited from

VisionLabeledModule.getInputShape

fromCustomModel()

static fromCustomModel<L>(modelSource, config, onDownloadProgress?): Promise<ObjectDetectionModule<L>>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:340

Creates an object detection instance with a user-provided model binary and label map. Use this when working with a custom-exported model that is not one of the built-in presets. Internally uses 'custom' as the model name for telemetry unless overridden.

Required model contract

The .pte model binary must expose a single forward method with the following interface:

Input: one float32 tensor of shape [1, 3, H, W] — a single RGB image, values in [0, 1] after optional per-channel normalization (pixel − mean) / std. H and W are read from the model's declared input shape at load time.

Outputs: exactly three float32 tensors, in this order:

Bounding boxes — flat [4·N] array of (x1, y1, x2, y2) coordinates in model-input pixel space, repeated for N detections.
Confidence scores — flat [N] array of values in [0, 1].
Class indices — flat [N] array of float32-encoded integer class indices (0-based, matching the order of entries in your labelMap).

Preprocessing (resize → normalize) and postprocessing (coordinate rescaling, threshold filtering, NMS) are handled by the native runtime — your model only needs to produce the raw detections above.

Type Parameters

L

L extends Readonly<Record<string, string | number>>

Parameters

modelSource

ResourceSource

A fetchable resource pointing to the model binary.

config

ObjectDetectionConfig<L>

A ObjectDetectionConfig object with the label map and optional preprocessing parameters.

onDownloadProgress?

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns

Promise<ObjectDetectionModule<L>>

A Promise resolving to an ObjectDetectionModule instance typed to the provided label map.

fromModelName()

static fromModelName<C>(namedSources, onDownloadProgress?): Promise<ObjectDetectionModule<ModelNameOf<C>>>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:106

Creates an object detection instance for a built-in model.

Type Parameters

C

C extends ObjectDetectionModelSources

Parameters

namedSources

C

A ObjectDetectionModelSources object specifying which model to load and where to fetch it from.

onDownloadProgress?

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns

Promise<ObjectDetectionModule<ModelNameOf<C>>>

A Promise resolving to an ObjectDetectionModule instance typed to the chosen model's label map.

Extends​

Type Parameters​

T​

Properties​

generateFromFrame()​

Parameters​

frameData​

args​

Returns​

See​

Inherited from​

labelMap​

Inherited from​

nativeModule​

Inherited from​

Accessors​

runOnFrame​

Get Signature​

Throws​

Returns​

Parameters​

frame​

isFrontCamera​

options?​

Returns​

Overrides​

Methods​

delete()​

Returns​

Inherited from​

forward()​

Parameters​

input​

options?​

Returns​

Throws​

Example​

Overrides​

forwardET()​

Parameters​

inputTensor​

Returns​

Inherited from​

getAvailableInputSizes()​

Returns​

Example​

getInputShape()​

Parameters​

methodName​

index​

Returns​

Inherited from​

fromCustomModel()​

Required model contract​

Type Parameters​

L​

Parameters​

modelSource​

config​

onDownloadProgress?​

Returns​

fromModelName()​

Type Parameters​

C​

Parameters​

namedSources​

onDownloadProgress?​

Returns​

Extends

Type Parameters

T

Properties

generateFromFrame()

Parameters

frameData

args

Returns

See

Inherited from

labelMap

Inherited from

nativeModule

Inherited from

Accessors

runOnFrame

Get Signature

Throws

Returns

Parameters

frame

isFrontCamera

options?

Returns

Overrides

Methods

delete()

Returns

Inherited from

forward()

Parameters

input

options?

Returns

Throws

Example

Overrides

forwardET()

Parameters

inputTensor

Returns

Inherited from

getAvailableInputSizes()

Returns

Example

getInputShape()

Parameters

methodName

index

Returns

Inherited from

fromCustomModel()

Required model contract

Type Parameters

L

Parameters

modelSource

config

onDownloadProgress?

Returns

fromModelName()

Type Parameters

C

Parameters

namedSources

onDownloadProgress?

Returns