useObjectDetection

Object detection is a computer vision technique that identifies and locates objects within images or video. It’s commonly used in applications like image recognition, video surveillance or autonomous driving. useObjectDetection is a hook that allows you to seamlessly integrate object detection into your React Native applications.

caution

It is recommended to use models provided by us, which are available at our Hugging Face repository. You can also use constants shipped with our library.

Reference

import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';

function App() {
  const ssdlite = useObjectDetection({
    modelSource: SSDLITE_320_MOBILENET_V3_LARGE, // alternatively, you can use require(...)
  });

  ...
  for (const detection of await ssdlite.forward("https://url-to-image.jpg")) {
    console.log("Bounding box: ", detection.bbox);
    console.log("Bounding label: ", detection.label);
    console.log("Bounding score: ", detection.score);
  }
  ...
}

Type definitions

interface Bbox {
  x1: number;
  x2: number;
  y1: number;
  y2: number;
}

interface Detection {
  bbox: Bbox;
  label: keyof typeof CocoLabel;
  score: number;
}

interface ObjectDetectionModule {
  error: string | null;
  isReady: boolean;
  isGenerating: boolean;
  forward: (input: string) => Promise<Detection[]>;
}

Arguments

modelSource

A string that specifies the path to the model file. You can download the model from our HuggingFace repository. For more information on that topic, you can check out the Loading models page.

Returns

The hook returns an object with the following properties:

Field	Type	Description
forward	(input: string) => Promise<Detection[]>	A function that accepts an image (url, b64) and returns an array of Detection objects.
error	string | null	Contains the error message if the model loading failed.
isGenerating	boolean	Indicates whether the model is currently processing an inference.
isReady	boolean	Indicates whether the model has successfully loaded and is ready for inference.

Running the model

To run the model, you can use the forward method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns an array of Detection objects. Each object contains coordinates of the bounding box, the label of the detected object, and the confidence score. For more information, please refer to the reference or type definitions.

Detection object

The detection object is specified as follows:

interface Bbox {
  x1: number;
  y1: number;
  x2: number;
  y2: number;
}

interface Detection {
  bbox: Bbox;
  label: keyof typeof CocoLabels;
  score: number;
}

The bbox property contains information about the bounding box of detected objects. It is represented as two points: one at the bottom-left corner of the bounding box (x1, y1) and the other at the top-right corner (x2, y2). The label property contains the name of the detected object, which corresponds to one of the CocoLabels. The score represents the confidence score of the detected object.

Example

import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';

function App() {
  const ssdlite = useObjectDetection({
    modelSource: SSDLITE_320_MOBILENET_V3_LARGE,
  });

  const runModel = async () => {
    const detections = await ssdlite.forward("https://url-to-image.jpg");
    for (const detection of detections) {
      console.log("Bounding box: ", detection.bbox);
      console.log("Bounding label: ", detection.label);
      console.log("Bounding score: ", detection.score);
    }
  }
}

Supported models

Model	Number of classes	Class list
SSDLite320 MobileNetV3 Large	91	COCO