ImageSegmentationModule

TypeScript API implementation of the useImageSegmentation hook.

API Reference

• For detailed API Reference for ImageSegmentationModule see: ImageSegmentationModule API Reference.
• For all image segmentation models available out-of-the-box in React Native ExecuTorch see: Image Segmentation Models.

High Level Overview

import {
  ImageSegmentationModule,
  DEEPLAB_V3_RESNET50,
} from 'react-native-executorch';

const imageUri = 'path/to/image.png';

// Creating an instance
const imageSegmentationModule = new ImageSegmentationModule();

// Loading the model
await imageSegmentationModule.load(DEEPLAB_V3_RESNET50);

// Running the model
const outputDict = await imageSegmentationModule.forward(imageUri);

Methods

All methods of ImageSegmentationModule are explained in details here: ImageSegmentationModule API Reference

Loading the model

To initialize the module, create an instance and call the load method with the following parameters:

• model - Object containing:

  • modelSource - Location of the used model.

• onDownloadProgressCallback - Callback to track download progress.

This method returns a promise, which can resolve to an error or void.

For more information on loading resources, take a look at loading models page.

Running the model

To run the model, you can use the forward method on the module object. It accepts three arguments: a required image, an optional list of classes, and an optional flag whether to resize the output to the original dimensions.

• The image can be a remote URL, a local file URI, or a base64-encoded image.
• The classesOfInterest list contains classes for which to output the full results. By default the list is empty, and only the most probable classes are returned (essentially an arg max for each pixel). Look at DeeplabLabel enum for possible classes.
• The resize flag says whether the output will be rescaled back to the size of the image you put in. The default is false. The model runs inference on a scaled (probably smaller) version of your image (224x224 for the DEEPLAB_V3_RESNET50). If you choose to resize, the output will be number[] of size width * height of your original image.

warning

Setting resize to true will make forward slower.

forward returns a promise which can resolve either to an error or a dictionary containing number arrays with size depending on resize:

• For the key DeeplabLabel.ARGMAX the array contains for each pixel an integer corresponding to the class with the highest probability.
• For every other key from DeeplabLabel, if the label was included in classesOfInterest the dictionary will contain an array of floats corresponding to the probability of this class for every pixel.

Managing memory

The module is a regular JavaScript object, and as such its lifespan will be managed by the garbage collector. In most cases this should be enough, and you should not worry about freeing the memory of the module yourself, but in some cases you may want to release the memory occupied by the module before the garbage collector steps in. In this case use the method delete on the module object you will no longer use, and want to remove from the memory. Note that you cannot use forward after delete unless you load the module again.