Skip to main content
Version: Next

Class: ObjectDetectionModule<T>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:55

Generic object detection module with type-safe label maps.

Extends​

  • VisionLabeledModule<Detection<ResolveLabels<T>>[], ResolveLabels<T>>

Type Parameters​

T​

T extends ObjectDetectionModelName | LabelEnum

Either a built-in model name (e.g. 'ssdlite-320-mobilenet-v3-large') or a custom LabelEnum label map.

Properties​

generateFromFrame()​

generateFromFrame: (frameData, ...args) => any

Defined in: modules/BaseModule.ts:53

Process a camera frame directly for real-time inference.

This method is bound to a native JSI function after calling load(), making it worklet-compatible and safe to call from VisionCamera's frame processor thread.

Performance characteristics:

  • Zero-copy path: When using frame.getNativeBuffer() from VisionCamera v5, frame data is accessed directly without copying (fastest, recommended).
  • Copy path: When using frame.toArrayBuffer(), pixel data is copied from native to JS, then accessed from native code (slower, fallback).

Usage with VisionCamera:

const frameOutput = useFrameOutput({
  pixelFormat: 'rgb',
  onFrame(frame) {
    'worklet';
    // Zero-copy approach (recommended)
    const nativeBuffer = frame.getNativeBuffer();
    const result = model.generateFromFrame(
      { nativeBuffer: nativeBuffer.pointer, width: frame.width, height: frame.height },
      ...args
    );
    nativeBuffer.release();
    frame.dispose();
  }
});

Parameters​

frameData​

Frame

Frame data object with either nativeBuffer (zero-copy) or data (ArrayBuffer)

args​

...any[]

Additional model-specific arguments (e.g., threshold, options)

Returns​

any

Model-specific output (e.g., detections, classifications, embeddings)

See​

Frame for frame data format details

Inherited from​

VisionLabeledModule.generateFromFrame

labelMap​

protected readonly labelMap: ResolveLabels

Defined in: modules/computer_vision/VisionLabeledModule.ts:42

Inherited from​

VisionLabeledModule.labelMap

nativeModule​

nativeModule: any = null

Defined in: modules/BaseModule.ts:16

Internal

Native module instance (JSI Host Object)

Inherited from​

VisionLabeledModule.nativeModule

Accessors​

runOnFrame​

Get Signature​

get runOnFrame(): (frame, ...args) => TOutput | null

Defined in: modules/computer_vision/VisionModule.ts:60

Synchronous worklet function for real-time VisionCamera frame processing.

Only available after the model is loaded. Returns null if not loaded.

Use this for VisionCamera frame processing in worklets. For async processing, use forward() instead.

Example​
const model = new ClassificationModule();
await model.load({ modelSource: MODEL });

// Use the functional form of setState to store the worklet — passing it
// directly would cause React to invoke it immediately as an updater fn.
const [runOnFrame, setRunOnFrame] = useState(null);
setRunOnFrame(() => model.runOnFrame);

const frameOutput = useFrameOutput({
  onFrame(frame) {
    'worklet';
    if (!runOnFrame) return;
    const result = runOnFrame(frame, isFrontCamera);
    frame.dispose();
  }
});
Returns​

(frame, ...args) => TOutput | null

A worklet function for frame processing, or null if the model is not loaded.

Inherited from​

VisionLabeledModule.runOnFrame

Methods​

delete()​

delete(): void

Defined in: modules/BaseModule.ts:81

Unloads the model from memory and releases native resources.

Always call this method when you're done with a model to prevent memory leaks.

Returns​

void

Inherited from​

VisionLabeledModule.delete

forward()​

forward(input, detectionThreshold?): Promise<Detection<ResolveLabels<T, { rf-detr-nano: { labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; }>>[]>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:104

Executes the model's forward pass to detect objects within the provided image.

Parameters​

input​

A string image source (file path, URI, or Base64) or a PixelData object.

string | PixelData

detectionThreshold?​

number = 0.7

Minimum confidence score for a detection to be included. Default is 0.7.

Returns​

Promise<Detection<ResolveLabels<T, { rf-detr-nano: { labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; }>>[]>

A Promise resolving to an array of Detection objects.

Overrides​

VisionLabeledModule.forward

forwardET()​

protected forwardET(inputTensor): Promise<TensorPtr[]>

Defined in: modules/BaseModule.ts:62

Internal

Runs the model's forward method with the given input tensors. It returns the output tensors that mimic the structure of output from ExecuTorch.

Parameters​

inputTensor​

TensorPtr[]

Array of input tensors.

Returns​

Promise<TensorPtr[]>

Array of output tensors.

Inherited from​

VisionLabeledModule.forwardET

getInputShape()​

getInputShape(methodName, index): Promise<number[]>

Defined in: modules/BaseModule.ts:72

Gets the input shape for a given method and index.

Parameters​

methodName​

string

method name

index​

number

index of the argument which shape is requested

Returns​

Promise<number[]>

The input shape as an array of numbers.

Inherited from​

VisionLabeledModule.getInputShape

fromCustomModel()​

static fromCustomModel<L>(modelSource, config, onDownloadProgress?): Promise<ObjectDetectionModule<L>>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:139

Creates an object detection instance with a user-provided model binary and label map. Use this when working with a custom-exported model that is not one of the built-in presets. Internally uses 'custom' as the model name for telemetry unless overridden.

Required model contract​

The .pte model binary must expose a single forward method with the following interface:

Input: one float32 tensor of shape [1, 3, H, W] — a single RGB image, values in [0, 1] after optional per-channel normalization (pixel − mean) / std. H and W are read from the model's declared input shape at load time.

Outputs: exactly three float32 tensors, in this order:

  1. Bounding boxes — flat [4·N] array of (x1, y1, x2, y2) coordinates in model-input pixel space, repeated for N detections.
  2. Confidence scores — flat [N] array of values in [0, 1].
  3. Class indices — flat [N] array of float32-encoded integer class indices (0-based, matching the order of entries in your labelMap).

Preprocessing (resize → normalize) and postprocessing (coordinate rescaling, threshold filtering, NMS) are handled by the native runtime — your model only needs to produce the raw detections above.

Type Parameters​

L​

L extends Readonly<Record<string, string | number>>

Parameters​

modelSource​

ResourceSource

A fetchable resource pointing to the model binary.

config​

ObjectDetectionConfig<L>

A ObjectDetectionConfig object with the label map and optional preprocessing parameters.

onDownloadProgress?​

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns​

Promise<ObjectDetectionModule<L>>

A Promise resolving to an ObjectDetectionModule instance typed to the provided label map.

fromModelName()​

static fromModelName<C>(namedSources, onDownloadProgress?): Promise<ObjectDetectionModule<ModelNameOf<C>>>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:68

Creates an object detection instance for a built-in model.

Type Parameters​

C​

C extends ObjectDetectionModelSources

Parameters​

namedSources​

C

A ObjectDetectionModelSources object specifying which model to load and where to fetch it from.

onDownloadProgress?​

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns​

Promise<ObjectDetectionModule<ModelNameOf<C>>>

A Promise resolving to an ObjectDetectionModule instance typed to the chosen model's label map.

We are Software Mansion.

Hire us