Text to Image

Text-to-image diffusion models generate photorealistic and artistic images directly from natural language descriptive prompts.

The library ships with SDXS-512 (Stable Diffusion eXtreme Speed) based on DreamShaper. Through architectural distillation, SDXS collapses multi-step denoising into a fast, single-step latent diffusion pipeline capable of generating 512x512 images completely on-device without cloud GPUs.

iOS	Android

Quick Start

The useTextToImage hook manages model downloading, CLIP tokenizer loading, and lifecycle:

import { models, useTextToImage } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function MyComponent() {
  const generator = useTextToImage(models.textToImage.SDXS_512_DREAMSHAPER.DEFAULT);

  // Hook state:
  // generator.isReady          — true once model and tokenizer are downloaded and loaded
  // generator.downloadProgress — 0 to 100 download progress
  // generator.error            — Error instance if download or load failed
  // generator.resource         — resolved config with all URLs replaced by local file paths

  const handleGenerate = async (prompt: string) => {
    if (!generator.isReady || !generator.generate) return;

    // Run inference on background thread (optional seed for deterministic output)
    const imageBuffer: ImageBuffer = await generator.generate(prompt, 42);
    console.log('Generated image:', imageBuffer.width, imageBuffer.height);
  };

  // Trigger handleGenerate on submit from a prompt input or button press
}

Full Interactive Example in Gallery App

See src/app/(screens)/text-to-image.tsx in the React Native ExecuTorch Gallery for a complete, runnable screen with prompt suggestions, generation progress, and Skia canvas rendering.

Output Format

generate() returns an ImageBuffer object with uncompressed 512x512 RGBA pixel bytes:

type ImageBuffer = {
  readonly width: 512;
  readonly height: 512;
  readonly format: 'rgba';
  readonly data: Uint8Array;
};

You can render the output directly to screen using React Native Skia, convert it into canvas textures, or pipe it into subsequent visual processing pipelines.

Determinism & Seeds

generate(prompt, seed) accepts an optional integer seed parameter:

• With a seed (e.g. generate("sunset over ocean", 123)): Reproduces the exact same image output deterministically.
• Without a seed (e.g. generate("sunset over ocean")): Uses a time-based random seed to produce a fresh variation on each execution.

Imperative API

For background generation jobs, headless services, or manual lifecycle management outside React components, create the generator using createSdxsTextToImage:

import { createSdxsTextToImage, download, models } from 'react-native-executorch';

// Download and cache model assets before creating the pipeline
const model = await download(models.textToImage.SDXS_512_DREAMSHAPER.DEFAULT);
const generator = await createSdxsTextToImage(model);

try {
  const imageBuffer = await generator.generate(
    'A serene mountain lake at sunrise, photorealistic, 8k',
    100
  );
  console.log('Generated image bytes:', imageBuffer.data.byteLength);
} finally {
  // Always release native resources when finished
  generator.dispose();
}

Synchronous Execution

For synchronous worklet execution contexts, createSdxsTextToImage exposes a generateWorklet function that executes directly inside a worklet runtime without Promise scheduling overhead:

// Called synchronously inside a worklet runtime
const imageBuffer = generator.generateWorklet(prompt, seed);

See Worklets & Threading for details on worklet execution contexts and zero-copy host objects.

Available Models

The library provides ready-to-use text-to-image models from the Software Mansion HuggingFace Text to Image Collection, available in models.textToImage:

Model Family	Variants	Resolution	Size Range	Supported Backends	Notes
SDXS 512 DreamShaper	See	512x512	839.9 MB – 1.64 GB	XNNPACK (CPU), Core ML (Apple)	Single-step distilled latent diffusion for ultra-fast on-device image synthesis.

Using Custom Models

To use your own fine-tuned SDXS .pte model and CLIP tokenizer, pass a SdxsTextToImageModel configuration object to useTextToImage or createSdxsTextToImage:

const customGenerator = await createSdxsTextToImage({
  modelPath: 'https://example.com/my-sdxs.pte',
  tokenizerPath: 'https://example.com/tokenizer.json',
});

The pipeline automatically verifies that the model's exported methods (encode, denoise, decode) match its requirements. To prepare and export your own .pte model to match this pipeline, see Exporting Custom Models.

API Reference

Hooks & Pipelines

• useTextToImage() — React hook for text-to-image model downloading, state, and lifecycle.
• createSdxsTextToImage() — Imperative factory for SDXS text-to-image pipelines.

Types & Options

• SdxsTextToImage — Text-to-image generator runner interface (generate, generateWorklet).
• SdxsTextToImageModel — Model configuration spec with modelPath and tokenizerPath.
• ImageBuffer — Generated RGBA output image buffer structure.

Model Presets

• models.textToImage — Pre-configured text-to-image generation models registry.

Source Code

View the implementation on GitHub:

• src/extensions/cv/tasks/sdxsTextToImage.ts ↗