Skip to main content
Version: Next

Class: TextToImageModule

Defined in: modules/computer_vision/TextToImageModule.ts:15

Module for text-to-image generation tasks.

Extends​

  • BaseModule

Constructors​

Constructor​

new TextToImageModule(inferenceCallback?): TextToImageModule

Defined in: modules/computer_vision/TextToImageModule.ts:23

Creates a new instance of TextToImageModule with optional callback on inference step.

Parameters​

inferenceCallback?​

(stepIdx) => void

Optional callback function that receives the current step index during inference.

Returns​

TextToImageModule

Overrides​

BaseModule.constructor

Properties​

generateFromFrame()​

generateFromFrame: (frameData, ...args) => any

Defined in: modules/BaseModule.ts:56

Process a camera frame directly for real-time inference.

This method is bound to a native JSI function after calling load(), making it worklet-compatible and safe to call from VisionCamera's frame processor thread.

Performance characteristics:

  • Zero-copy path: When using frame.getNativeBuffer() from VisionCamera v5, frame data is accessed directly without copying (fastest, recommended).
  • Copy path: When using frame.toArrayBuffer(), pixel data is copied from native to JS, then accessed from native code (slower, fallback).

Usage with VisionCamera:

const frameOutput = useFrameOutput({
  pixelFormat: 'rgb',
  onFrame(frame) {
    'worklet';
    // Zero-copy approach (recommended)
    const nativeBuffer = frame.getNativeBuffer();
    const result = model.generateFromFrame(
      { nativeBuffer: nativeBuffer.pointer, width: frame.width, height: frame.height },
      ...args
    );
    nativeBuffer.release();
    frame.dispose();
  }
});

Parameters​

frameData​

Frame

Frame data object with either nativeBuffer (zero-copy) or data (ArrayBuffer)

args​

...any[]

Additional model-specific arguments (e.g., threshold, options)

Returns​

any

Model-specific output (e.g., detections, classifications, embeddings)

See​

Frame for frame data format details

Inherited from​

BaseModule.generateFromFrame

nativeModule​

nativeModule: any = null

Defined in: modules/BaseModule.ts:17

Internal

Native module instance (JSI Host Object)

Inherited from​

BaseModule.nativeModule

Methods​

delete()​

delete(): void

Defined in: modules/BaseModule.ts:100

Unloads the model from memory and releases native resources.

Always call this method when you're done with a model to prevent memory leaks.

Returns​

void

Inherited from​

BaseModule.delete

forward()​

forward(input, imageSize?, numSteps?, seed?): Promise<string>

Defined in: modules/computer_vision/TextToImageModule.ts:106

Runs the model to generate an image described by input, and conditioned by seed, performing numSteps inference steps. The resulting image, with dimensions imageSize×imageSize pixels, is returned as a base64-encoded string.

Parameters​

input​

string

The text prompt to generate the image from.

imageSize?​

number = 512

The desired width and height of the output image in pixels.

numSteps?​

number = 5

The number of inference steps to perform.

seed?​

number

An optional seed for random number generation to ensure reproducibility.

Returns​

Promise<string>

A Base64-encoded string representing the generated PNG image.

forwardET()​

protected forwardET(inputTensor): Promise<TensorPtr[]>

Defined in: modules/BaseModule.ts:80

Internal

Runs the model's forward method with the given input tensors. It returns the output tensors that mimic the structure of output from ExecuTorch.

Parameters​

inputTensor​

TensorPtr[]

Array of input tensors.

Returns​

Promise<TensorPtr[]>

Array of output tensors.

Inherited from​

BaseModule.forwardET

getInputShape()​

getInputShape(methodName, index): Promise<number[]>

Defined in: modules/BaseModule.ts:91

Gets the input shape for a given method and index.

Parameters​

methodName​

string

method name

index​

number

index of the argument which shape is requested

Returns​

Promise<number[]>

The input shape as an array of numbers.

Inherited from​

BaseModule.getInputShape

interrupt()​

interrupt(): void

Defined in: modules/computer_vision/TextToImageModule.ts:138

Interrupts model generation. The model is stopped in the nearest step.

Returns​

void

load()​

load(model, onDownloadProgressCallback?): Promise<void>

Defined in: modules/computer_vision/TextToImageModule.ts:36

Loads the model from specified resources.

Parameters​

model​

Object containing sources for tokenizer, scheduler, encoder, unet, and decoder.

decoderSource​

ResourceSource

encoderSource​

ResourceSource

schedulerSource​

ResourceSource

tokenizerSource​

ResourceSource

unetSource​

ResourceSource

onDownloadProgressCallback?​

(progress) => void

Optional callback to monitor download progress.

Returns​

Promise<void>

Overrides​

BaseModule.load

We are Software Mansion.

Hire us