Object Detection

Object detection locates and classifies multiple objects within an image. For every detected item, the model predicts its category label, confidence score, and exact bounding box coordinates scaled to the original image dimensions.

Unlike image classification (which predicts a single label for the entire scene), object detection tells you both what objects are present and where they are located. It is used for real-time camera tracking, retail item recognition, document scanning, robotics, and augmented reality.

iOS	Android

Quick Start

The useObjectDetector hook manages model downloading, initialization, and lifecycle:

import { models, useObjectDetector } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function MyComponent() {
  const detector = useObjectDetector(models.objectDetection.SSDLITE320_MOBILENET_V3_LARGE.DEFAULT);

  // Hook state:
  // detector.isReady          — true once model is downloaded and loaded in memory
  // detector.downloadProgress — 0 to 100 download progress
  // detector.error            — Error instance if download or load failed
  // detector.resource         — resolved config with all URLs replaced by local file paths

  const handleDetect = async (imageBuffer: ImageBuffer) => {
    if (!detector.isReady || !detector.detectObjects) return;

    // Run inference on background thread
    const detections = await detector.detectObjects(imageBuffer, {
      confidenceThreshold: 0.5,
      iouThreshold: 0.55,
    });
    console.log('Detected objects:', detections);
  };

  // Trigger handleDetect from an image picker, button press, or camera frame
}

Full Interactive Example in Gallery App

See src/app/(screens)/object-detection.tsx in the React Native ExecuTorch Gallery for a complete, runnable screen with photo picker, bounding box overlays, and latency tracking.

Output Format

detectObjects() returns an array of ObjectDetection objects:

type ObjectDetection<F extends BoxFormat = 'xyxy', L = string> = {
  /** Scaled bounding box coordinates matching the input image dimensions */
  readonly box: BoundingBox<F>;
  /** Predicted object class label */
  readonly label: L;
  /** Confidence score of the detection (between 0.0 and 1.0) */
  readonly confidence: number;
};

For 'xyxy' format (default), box contains pixel coordinates:

[
  {
    box: { format: 'xyxy', xmin: 34.5, ymin: 112.0, xmax: 240.2, ymax: 380.7 },
    label: 'dog',
    confidence: 0.89,
  },
  {
    box: { format: 'xyxy', xmin: 310.0, ymin: 85.3, xmax: 520.1, ymax: 410.0 },
    label: 'person',
    confidence: 0.94,
  },
];

Configuration & Options

Pass a DetectObjectsOptions object to detectObjects() to override model defaults:

Option	Type	Default	Description
confidenceThreshold	number	Model default (e.g. 0.5)	Minimum confidence score for a box to be retained (0.0 to 1.0).
iouThreshold	number	Model default (e.g. 0.55)	Intersection over Union (IoU) threshold for Non-Maximum Suppression (NMS). Lower values suppress more overlapping boxes.

Imperative API

For background tasks or headless usage outside React components, create the detector using createObjectDetector:

import { createObjectDetector, download, models } from 'react-native-executorch';

// Download and cache model assets before creating the pipeline
const model = await download(models.objectDetection.SSDLITE320_MOBILENET_V3_LARGE.DEFAULT);
const detector = await createObjectDetector(model);

try {
  const detections = await detector.detectObjects(imageBuffer, {
    confidenceThreshold: 0.4,
  });
  console.log('Detections:', detections);
} finally {
  // Always release native resources when finished
  detector.dispose();
}

Synchronous Execution

For high-throughput loops like camera frame processors, createObjectDetector exposes a synchronous detectObjectsWorklet function. This runs directly on the worklet thread with zero Promise scheduling overhead:

// Called synchronously inside a VisionCamera frame processor on the UI worklet thread
const detections = detector.detectObjectsWorklet(frameBuffer, {
  confidenceThreshold: 0.5,
});

See Worklets & Threading for details on worklet execution contexts and zero-copy host objects.

Available Models

The library provides ready-to-use detectors from the Software Mansion HuggingFace Object Detection Collection, trained on the 80-category COCO dataset and available in models.objectDetection:

Model Family	Variants	Size Range	Supported Backends	Dataset / Vocabulary	Notes
SSDLite320 MobileNetV3	See	8.1 MB – 13.3 MB	XNNPACK (CPU), Core ML (Apple)	COCO_CLASSES (80 classes)	Ultra-lightweight detector with highest frame rates on low-end devices.
RF-DETR Nano	See	52.2 MB – 106.4 MB	XNNPACK (CPU), Core ML (Apple)	COCO_CLASSES (80 classes)	DINOv2-based detection transformer with superior small-object accuracy.
YOLO26	See	5.4 MB – 212.9 MB	XNNPACK (CPU), Core ML (Apple)	COCO_CLASSES_YOLO (80 classes)	Scalable real-time detection family across 384x384, 512x512, and 640x640 resolutions.

Using Custom Models

To use your own fine-tuned object detection .pte model, pass an ObjectDetectorModel configuration object to useObjectDetector or createObjectDetector:

const customDetector = await createObjectDetector({
  modelPath: 'https://example.com/my-detector.pte',
  modelOpts: {
    labels: ['hardhat', 'vest', 'boots'],
    boxFormat: 'xyxy',
    resizeMode: 'stretch',
    interpolation: 'linear',
    normalizeOpts: { alpha: 1 / 255.0, beta: 0.0 },
    defaultConfidenceThreshold: 0.4,
    defaultIouThreshold: 0.5,
  },
});

The pipeline automatically verifies that the model's exported input and output shapes match its requirements. To prepare and export your own .pte model to match this pipeline, see Exporting Custom Models.

API Reference

Hooks & Pipelines

• useObjectDetector() — React hook for object detector downloading, state, and lifecycle.
• createObjectDetector() — Imperative factory for object detector task pipelines.

Types & Options

• ObjectDetector — Object detector instance interface (detectObjects, detectObjectsWorklet).
• ObjectDetection — Single detection result with box, label, and confidence.
• DetectObjectsOptions — Inference options (confidenceThreshold, iouThreshold).
• ObjectDetectorModel — Object detector configuration spec.
• ObjectDetectorOptions — Options defining labels, box format, and normalization.
• BoundingBox — Generic bounding box structure.
• BoxFormat — Coordinate formats ('xyxy', 'xywh', 'cxcywh').
• ImageBuffer — Input image buffer structure.

Model Presets

• models.objectDetection — Pre-configured object detection models registry.

Source Code

View the implementation on GitHub:

• src/extensions/cv/tasks/objectDetection.ts ↗