Class: InstanceSegmentationModule<T>

Defined in: modules/computer_vision/InstanceSegmentationModule.ts:136

Generic instance segmentation module with type-safe label maps. Use a model name (e.g. 'yolo26n-seg') as the generic parameter for pre-configured models, or a custom label enum for custom configs.

Supported models (download from HuggingFace):

• yolo26n-seg, yolo26s-seg, yolo26m-seg, yolo26l-seg, yolo26x-seg - YOLO models with COCO labels (80 classes)
• rfdetr-nano-seg - RF-DETR Nano model with COCO labels (80 classes)

Example

const segmentation = await InstanceSegmentationModule.fromModelName({
  modelName: 'yolo26n-seg',
  modelSource: 'https://huggingface.co/.../yolo26n-seg.pte',
});

const results = await segmentation.forward('path/to/image.jpg', {
  confidenceThreshold: 0.5,
  iouThreshold: 0.45,
  maxInstances: 20,
  inputSize: 640,
});

Extends

• VisionLabeledModule<SegmentedInstance<ResolveLabels<T>>[], ResolveLabels<T>>

Type Parameters

T

T extends InstanceSegmentationModelName | LabelEnum

Either a pre-configured model name from InstanceSegmentationModelName or a custom label map conforming to LabelEnum.

Properties

generateFromFrame()

generateFromFrame: (frameData, ...args) => any

Defined in: modules/BaseModule.ts:53

Process a camera frame directly for real-time inference.

This method is bound to a native JSI function after calling load(), making it worklet-compatible and safe to call from VisionCamera's frame processor thread.

Performance characteristics:

• Zero-copy path: When using frame.getNativeBuffer() from VisionCamera v5, frame data is accessed directly without copying (fastest, recommended).
• Copy path: When using frame.toArrayBuffer(), pixel data is copied from native to JS, then accessed from native code (slower, fallback).

Usage with VisionCamera:

const frameOutput = useFrameOutput({
  pixelFormat: 'rgb',
  onFrame(frame) {
    'worklet';
    // Zero-copy approach (recommended)
    const nativeBuffer = frame.getNativeBuffer();
    const result = model.generateFromFrame(
      { nativeBuffer: nativeBuffer.pointer, width: frame.width, height: frame.height },
      ...args
    );
    nativeBuffer.release();
    frame.dispose();
  }
});

Parameters

frameData

Frame

Frame data object with either nativeBuffer (zero-copy) or data (ArrayBuffer)

args

...any[]

Additional model-specific arguments (e.g., threshold, options)

Returns

any

Model-specific output (e.g., detections, classifications, embeddings)

See

Frame for frame data format details

Inherited from

VisionLabeledModule.generateFromFrame

---

labelMap

protected readonly labelMap: ResolveLabels

Defined in: modules/computer_vision/VisionLabeledModule.ts:42

Inherited from

VisionLabeledModule.labelMap

---

nativeModule

nativeModule: any = null

Defined in: modules/BaseModule.ts:16

Internal

Native module instance (JSI Host Object)

Inherited from

VisionLabeledModule.nativeModule

Accessors

runOnFrame

Get Signature

get runOnFrame(): (frame, isFrontCamera, options?) => SegmentedInstance<ResolveLabels<T, { rfdetr-nano-seg: { availableInputSizes: undefined; defaultConfidenceThreshold: number; defaultInputSize: undefined; defaultIouThreshold: number; labelMap: typeof CocoLabel; postprocessorConfig: { applyNMS: true; }; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; yolo26l-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26m-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26n-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26s-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26x-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; }>>[] | null

Defined in: modules/computer_vision/InstanceSegmentationModule.ts:281

Override runOnFrame to add label mapping for VisionCamera integration. The parent's runOnFrame returns raw native results with class indices; this override maps them to label strings and provides an options-based API.

Returns

(frame, isFrontCamera, options?) => SegmentedInstance<ResolveLabels<T, { rfdetr-nano-seg: { availableInputSizes: undefined; defaultConfidenceThreshold: number; defaultInputSize: undefined; defaultIouThreshold: number; labelMap: typeof CocoLabel; postprocessorConfig: { applyNMS: true; }; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; yolo26l-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26m-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26n-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26s-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26x-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; }>>[] | null

A worklet function for VisionCamera frame processing, or null if the model is not loaded.

Overrides

VisionLabeledModule.runOnFrame

Methods

delete()

delete(): void

Defined in: modules/BaseModule.ts:81

Unloads the model from memory and releases native resources.

Always call this method when you're done with a model to prevent memory leaks.

Returns

void

Inherited from

VisionLabeledModule.delete

---

forward()

forward(input, options?): Promise<SegmentedInstance<ResolveLabels<T, { rfdetr-nano-seg: { availableInputSizes: undefined; defaultConfidenceThreshold: number; defaultInputSize: undefined; defaultIouThreshold: number; labelMap: typeof CocoLabel; postprocessorConfig: { applyNMS: true; }; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; yolo26l-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26m-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26n-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26s-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26x-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; }>>[]>

Defined in: modules/computer_vision/InstanceSegmentationModule.ts:383

Executes the model's forward pass to perform instance segmentation on the provided image.

Supports two input types:

1. String path/URI: File path, URL, or Base64-encoded string
2. PixelData: Raw pixel data from image libraries (e.g., NitroImage)

Parameters

input

Image source (string path or PixelData object)

string | PixelData

options?

InstanceSegmentationOptions<ResolveLabels<T, { rfdetr-nano-seg: { availableInputSizes: undefined; defaultConfidenceThreshold: number; defaultInputSize: undefined; defaultIouThreshold: number; labelMap: typeof CocoLabel; postprocessorConfig: { applyNMS: true; }; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; yolo26l-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26m-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26n-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26s-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26x-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; }>>

Optional configuration for the segmentation process. Includes confidenceThreshold, iouThreshold, maxInstances, classesOfInterest, returnMaskAtOriginalResolution, and inputSize.

Returns

Promise<SegmentedInstance<ResolveLabels<T, { rfdetr-nano-seg: { availableInputSizes: undefined; defaultConfidenceThreshold: number; defaultInputSize: undefined; defaultIouThreshold: number; labelMap: typeof CocoLabel; postprocessorConfig: { applyNMS: true; }; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; yolo26l-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26m-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26n-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26s-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; yolo26x-seg: { availableInputSizes: readonly [384, 512, 640]; defaultConfidenceThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; postprocessorConfig: { applyNMS: false; }; preprocessorConfig: undefined; }; }>>[]>

A Promise resolving to an array of SegmentedInstance objects with bbox, mask, maskWidth, maskHeight, label, score.

Throws

If the model is not loaded or if an invalid inputSize is provided.

Example

const results = await segmentation.forward('path/to/image.jpg', {
  confidenceThreshold: 0.6,
  iouThreshold: 0.5,
  maxInstances: 10,
  inputSize: 640,
  classesOfInterest: ['PERSON', 'CAR'],
  returnMaskAtOriginalResolution: true,
});

results.forEach((inst) => {
  console.log(`${inst.label}: ${(inst.score * 100).toFixed(1)}%`);
});

Overrides

VisionLabeledModule.forward

---

forwardET()

protected forwardET(inputTensor): Promise<TensorPtr[]>

Defined in: modules/BaseModule.ts:62

Internal

Runs the model's forward method with the given input tensors. It returns the output tensors that mimic the structure of output from ExecuTorch.

Parameters

inputTensor

TensorPtr[]

Array of input tensors.

Returns

Promise<TensorPtr[]>

Array of output tensors.

Inherited from

VisionLabeledModule.forwardET

---

getAvailableInputSizes()

getAvailableInputSizes(): readonly number[] | undefined

Defined in: modules/computer_vision/InstanceSegmentationModule.ts:271

Returns the available input sizes for this model, or undefined if the model accepts any size.

Returns

readonly number[] | undefined

An array of available input sizes, or undefined if not constrained.

Example

const sizes = segmentation.getAvailableInputSizes();
console.log(sizes); // [384, 512, 640] for YOLO models, or undefined for RF-DETR

---

getInputShape()

getInputShape(methodName, index): Promise<number[]>

Defined in: modules/BaseModule.ts:72

Gets the input shape for a given method and index.

Parameters

methodName

string

method name

index

number

index of the argument which shape is requested

Returns

Promise<number[]>

The input shape as an array of numbers.

Inherited from

VisionLabeledModule.getInputShape

---

fromCustomConfig()

static fromCustomConfig<L>(modelSource, config, onDownloadProgress?): Promise<InstanceSegmentationModule<L>>

Defined in: modules/computer_vision/InstanceSegmentationModule.ts:230

Creates an instance segmentation module with a user-provided label map and custom config. Use this when working with a custom-exported segmentation model that is not one of the pre-configured models.

Type Parameters

L

L extends Readonly<Record<string, string | number>>

Parameters

modelSource

ResourceSource

A fetchable resource pointing to the model binary.

config

InstanceSegmentationConfig<L>

A InstanceSegmentationConfig object with the label map and optional preprocessing parameters.

onDownloadProgress?

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns

Promise<InstanceSegmentationModule<L>>

A Promise resolving to an InstanceSegmentationModule instance typed to the provided label map.

Example

const MyLabels = { PERSON: 0, CAR: 1 } as const;
const segmentation = await InstanceSegmentationModule.fromCustomConfig(
  'https://huggingface.co/.../custom_model.pte',
  {
    labelMap: MyLabels,
    availableInputSizes: [640],
    defaultInputSize: 640,
    defaultConfidenceThreshold: 0.5,
    defaultIouThreshold: 0.45,
    postprocessorConfig: { applyNMS: true },
  },
);

---

fromModelName()

static fromModelName<C>(config, onDownloadProgress?): Promise<InstanceSegmentationModule<InstanceModelNameOf<C>>>

Defined in: modules/computer_vision/InstanceSegmentationModule.ts:173

Creates an instance segmentation module for a pre-configured model. The config object is discriminated by modelName — each model can require different fields.

Type Parameters

C

C extends InstanceSegmentationModelSources

Parameters

config

C

A InstanceSegmentationModelSources object specifying which model to load and where to fetch it from.

onDownloadProgress?

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns

Promise<InstanceSegmentationModule<InstanceModelNameOf<C>>>

A Promise resolving to an InstanceSegmentationModule instance typed to the chosen model's label map.

Example

const segmentation = await InstanceSegmentationModule.fromModelName({
  modelName: 'yolo26n-seg',
  modelSource: 'https://huggingface.co/.../yolo26n-seg.pte',
});