Semantic Segmentation

Semantic segmentation classifies every individual pixel of an input image into a designated category label (e.g. background, person, vehicle, dog). The pipeline produces a pixel-aligned segmentation mask matching the input dimensions.

Unlike object detection (which outputs rectangular bounding boxes), semantic segmentation delivers precise pixel boundaries. It powers photo portrait effects, background blur/replacement, scene parsing, medical imaging, and autonomous navigation.

iOS	Android

Quick Start

The useSemanticSegmenter hook manages model downloading, initialization, and lifecycle:

import { models, useSemanticSegmenter } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function MyComponent() {
  const segmenter = useSemanticSegmenter(models.semanticSegmentation.DEEPLAB_V3_RESNET50.DEFAULT);

  // Hook state:
  // segmenter.isReady          — true once model is downloaded and loaded in memory
  // segmenter.downloadProgress — 0 to 100 download progress
  // segmenter.error            — Error instance if download or load failed
  // segmenter.resource         — resolved config with all URLs replaced by local file paths

  const handleSegment = async (imageBuffer: ImageBuffer) => {
    if (!segmenter.isReady || !segmenter.segment) return;

    // Run inference on background thread
    const result = await segmenter.segment(imageBuffer);
    console.log('Output mask buffer:', result.buffer);
  };

  // Trigger handleSegment from an image picker, button press, or camera frame
}

Full Interactive Example in Gallery App

See src/app/(screens)/semantic-segmentation.tsx in the React Native ExecuTorch Gallery for a complete, runnable screen with photo picker, custom colormap blending, and latency tracking.

Output Format

segment() returns a SemanticSegmentationResult object:

type SemanticSegmentationResult<L extends PropertyKey = string> = {
  /** Output RGBA image buffer containing the colored segmentation mask */
  readonly buffer: ImageBuffer;
  /** Applied color map mapping each class label to its [R, G, B, A] tuple */
  readonly colormap?: ColorMap<L>;
};

Color Mapping Behavior

• Multi-class models (e.g. DEEPLAB_V3, LRASPP): Performs an argmax over the class logits per pixel, then maps each class index to its corresponding [R, G, B, A] color tuple. The returned colormap contains the full active label-to-color mapping.
• Single-class / binary models (e.g. SELFIE_SEGMENTATION): Applies a sigmoid activation to the single output channel, scales probabilities to pixel intensity values (0–255), and returns an RGBA mask. No color map is applied, and colormap is undefined.

Configuration & Color Maps

Pass an optional partial ColorMap object to segment() to customize how categories are colored:

// Custom RGBA colors: [R, G, B, A] (values 0 - 255)
const result = await segmenter.segment(imageBuffer, {
  person: [255, 0, 0, 180], // Translucent red for person
  background: [0, 0, 0, 0], // Fully transparent for background
});

When omitted, multi-class models automatically generate high-contrast distinct colors with the first class (typically background) defaulting to transparent [0, 0, 0, 0]. If a partial map is provided, any labels omitted from it will default to being rendered as fully transparent.

Imperative API

For background processing, headless pipelines, or manual lifecycle management outside React components, create the segmenter using createSemanticSegmenter:

import { createSemanticSegmenter, download, models } from 'react-native-executorch';

// Download and cache model assets before creating the pipeline
const model = await download(models.semanticSegmentation.DEEPLAB_V3_RESNET50.DEFAULT);
const segmenter = await createSemanticSegmenter(model);

try {
  const result = await segmenter.segment(imageBuffer);
  console.log('Generated mask dimensions:', result.buffer.width, result.buffer.height);
} finally {
  // Always release native resources when finished
  segmenter.dispose();
}

Synchronous Execution

For high-throughput loops like live camera background removal or portrait mode effects, createSemanticSegmenter exposes a synchronous segmentWorklet function. This runs directly on the worklet thread with zero Promise scheduling overhead:

// Called synchronously inside a VisionCamera frame processor on the UI worklet thread
const result = segmenter.segmentWorklet(frameBuffer);

See Worklets & Threading for details on worklet execution contexts and zero-copy host objects.

Available Models

The library provides ready-to-use segmentation models from the Software Mansion HuggingFace Semantic Segmentation Collection, accessible via models.semanticSegmentation:

Model Family	Variants	Classes / Labels	Size Range	Supported Backends	Notes
Selfie Segmentation	Portrait, Landscape	Person / Background	0.5 MB – 0.6 MB	XNNPACK (CPU), Core ML (Apple)	Real-time front-camera portrait background replacement and blur effects.
LRASPP MobileNetV3	See	PASCAL_VOC_LABELS (21 classes)	3.4 MB – 12.3 MB	XNNPACK (CPU), Core ML (Apple)	Lightweight multi-class scene segmentation with low CPU overhead.
DeepLabV3	ResNet50, ResNet101, MobileNetV3	PASCAL_VOC_LABELS (21 classes)	40.4 MB – 223.6 MB	XNNPACK (CPU), Core ML (Apple)	High-fidelity dense pixel classification for complex scenes.
FCN	ResNet50, ResNet101	PASCAL_VOC_LABELS (21 classes)	34.0 MB – 198.1 MB	XNNPACK (CPU), Core ML (Apple)	Fully Convolutional Networks baseline for dense multi-class parsing.

Using Custom Models

To use your own fine-tuned semantic segmentation .pte model, pass a SemanticSegmenterModel configuration object to useSemanticSegmenter or createSemanticSegmenter:

const customSegmenter = await createSemanticSegmenter({
  modelPath: 'https://example.com/my-segmentation.pte',
  modelOpts: {
    labels: ['background', 'road', 'sidewalk', 'building'],
    resizeMode: 'stretch',
    interpolation: 'linear',
    outInterpolation: 'lanczos',
    normalizeOpts: { alpha: 1 / 255.0, beta: 0.0 },
  },
});

The pipeline automatically verifies that the model's exported input and output shapes match its requirements. To prepare and export your own .pte model to match this pipeline, see Exporting Custom Models.

API Reference

Hooks & Pipelines

• useSemanticSegmenter() — React hook for semantic segmenter downloading, state, and lifecycle.
• createSemanticSegmenter() — Imperative factory for semantic segmentation pipelines.

Types & Options

• SemanticSegmenter — Semantic segmenter runner interface (segment, segmentWorklet).
• SemanticSegmentationResult — Output structure containing buffer and colormap.
• ColorMap — Map of label names to [R, G, B, A] tuples.
• SemanticSegmenterModel — Model configuration spec for semantic segmenter pipelines.
• SemanticSegmenterOptions — Options defining labels, interpolation, and normalization.
• ImageBuffer — Input and output image buffer structure.

Model Presets & Constants

• models.semanticSegmentation — Pre-configured semantic segmentation models registry.
• PASCAL_VOC_LABELS — List of 21 standard Pascal VOC class labels.

Source Code

View the implementation on GitHub:

• src/extensions/cv/tasks/semanticSegmentation.ts ↗