Image Embeddings

Image embedding models extract high-dimensional semantic feature vectors (embeddings) from raw images. When paired with multimodal models like OpenAI CLIP (Contrastive Language-Image Pretraining) and Text Embeddings, image and text embeddings share the same joint vector space.

This enables on-device cross-modal photo search (finding pictures with natural language queries), zero-shot image classification, visual similarity clustering, and vector search against local SQLite vector stores — all computed entirely on-device without network latency or cloud costs.

iOS	Android

Quick Start

The useImageEmbedder hook manages model downloading, initialization, and lifecycle:

import { models, useImageEmbedder } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function MyComponent() {
  const imageEmbedder = useImageEmbedder(models.imageEmbeddings.CLIP_VIT_BASE_PATCH32.DEFAULT);

  // Hook state:
  // imageEmbedder.isReady          — true once model is downloaded and loaded in memory
  // imageEmbedder.downloadProgress — 0 to 100 download progress
  // imageEmbedder.error            — Error instance if download or load failed
  // imageEmbedder.resource         — resolved config with all URLs replaced by local file paths

  const handleEmbed = async (imageBuffer: ImageBuffer) => {
    if (!imageEmbedder.isReady || !imageEmbedder.embed) return;

    // Run inference on background thread
    const vector = await imageEmbedder.embed(imageBuffer);
    console.log('Embedding dimension:', vector.length); // 512
  };

  // Trigger handleEmbed from an image picker, button press, or camera frame
}

Full Interactive Example in Gallery App

See src/app/(screens)/image-embeddings.tsx in the React Native ExecuTorch Gallery for a complete, runnable screen combining image and text embeddings for real-time zero-shot photo ranking.

Output Format

embed() returns a 1D Float32Array containing the normalized feature vector:

// Float32Array of length D (e.g. 512 for CLIP ViT-B/32)
const vector: Float32Array = await imageEmbedder.embed(imageBuffer);

Cross-Modal Similarity Matching

To compute the cosine similarity between an image embedding and a text query embedding produced by useTextEmbedder, compute their dot product:

function cosineSimilarity(a: Float32Array, b: Float32Array): number {
  let sum = 0;
  for (let i = 0; i < a.length; i++) {
    sum += a[i] * b[i];
  }
  return sum;
}

// Compare image vector with query text vector
const score = cosineSimilarity(imageVector, textVector);
console.log('Match similarity score:', score);

Imperative API

For background indexing, SQLite vector ingestion, or manual lifecycle management outside React components, create the embedder using createImageEmbedder:

import { createImageEmbedder, download, models } from 'react-native-executorch';

// Download and cache model assets before creating the pipeline
const model = await download(models.imageEmbeddings.CLIP_VIT_BASE_PATCH32.DEFAULT);
const embedder = await createImageEmbedder(model);

try {
  const vector = await embedder.embed(imageBuffer);
  console.log('Generated vector:', vector.slice(0, 5));
} finally {
  // Always release native resources when finished
  embedder.dispose();
}

Synchronous Execution

For high-throughput loops or real-time camera feature extraction, createImageEmbedder exposes a synchronous embedWorklet function. This runs directly on the worklet thread with zero Promise scheduling overhead:

// Called synchronously inside a worklet runtime on the UI thread
const vector = embedder.embedWorklet(frameBuffer);

See Worklets & Threading for details on worklet execution contexts and zero-copy host objects.

Available Models

The library provides ready-to-use vision encoders from the Software Mansion HuggingFace Image Embeddings Collection, available in models.imageEmbeddings:

Model Family	Variants	Output Dim	Size Range	Supported Backends	Notes
CLIP ViT-B/32 Vision	See	512	93.7 MB – 335.3 MB	XNNPACK (CPU), Core ML (Apple), MLX (Apple), Vulkan (Android)	Joint image-text semantic search, image clustering, and zero-shot categorization.

Using Custom Models

To use your own fine-tuned vision encoder .pte model, pass an ImageEmbedderModel configuration object to useImageEmbedder or createImageEmbedder:

const customEmbedder = await createImageEmbedder({
  modelPath: 'https://example.com/my-vision-encoder.pte',
  modelOpts: {
    resizeMode: 'stretch',
    interpolation: 'linear',
    normalizeOpts: { alpha: 1 / 255.0, beta: 0.0 },
  },
});

The pipeline automatically verifies that the model's exported input and output shapes match its requirements. To prepare and export your own .pte model to match this pipeline, see Exporting Custom Models.

API Reference

Hooks & Pipelines

• useImageEmbedder() — React hook for vision embedding model downloading, state, and lifecycle.
• createImageEmbedder() — Imperative factory for vision embedding pipelines.
• useTextEmbedder() — React hook for text embedding models to pair with vision encoders.

Types & Options

• ImageEmbedder — Image embedder runner interface (embed, embedWorklet).
• ImageEmbedderModel — Model configuration spec for vision embedders.
• ImagePreprocessorOptions — Options defining normalization, interpolation, and resize modes.
• ImageBuffer — Input image buffer structure.

Model Presets

• models.imageEmbeddings — Pre-configured vision encoder models registry.

Source Code

View the implementation on GitHub:

• src/extensions/cv/tasks/imageEmbedding.ts ↗