Skip to main content
Version: 0.10.0

LLM Chat & Text Generation

The LLM extension lets you run generative Large Language Models (LLMs) and Vision-Language Models (VLMs) directly on user devices with real-time token streaming, complete privacy, and full offline support. Depending on what you are building, you can choose between two levels of control:

iOSAndroid

Quick Start​

The useLLMChatSession hook handles remote model downloading, caching, tokenizer setup, and conversational state in a single React hook:

import { useState } from 'react';
import { models, useLLMChatSession } from 'react-native-executorch';

function MyChatComponent() {
  const [streamingText, setStreamingText] = useState('');

  const session = useLLMChatSession(models.llm.LFM2_5_1_2B.DEFAULT, {
    initialMessages: [{ role: 'system', content: 'You are a helpful on-device assistant.' }],
    generationConfig: {
      temperature: 0.2,
      maxNewTokens: 512,
    },
  });

  // Hook state:
  // session.isReady          — true once model weights and tokenizer are loaded in memory
  // session.downloadProgress — 0 to 100 download progress
  // session.error            — Error instance if download or load failed
  // session.resource         — resolved config with all URLs replaced by local file paths

  const handleSend = async (userPrompt: string) => {
    if (!session.isReady || !session.sendMessage) return;

    setStreamingText('');

    // Stream tokens as they are decoded
    const turn = await session.sendMessage(userPrompt, (token) => {
      setStreamingText((prev) => prev + token);
    });

    console.log('New messages added this turn:', turn.messages);
    console.log('Turn generation statistics:', turn.stats);
  };

  // Trigger handleSend on submit from a prompt input or chat screen
}
Full Interactive Example in Gallery App

See src/app/(screens)/llm-chat.tsx in the React Native ExecuTorch Gallery for a complete, runnable chat UI with token streaming, token/sec benchmarking, and message history.

Understanding the Output & Turn Result​

When you call sendMessage(), the promise resolves to an LLMChatTurnResult describing what happened during that turn:

type LLMChatTurnResult = {
  /**
   * The new messages added to conversation history during this turn.
   * Includes the user prompt, any assistant tool calls, tool responses,
   * and the final assistant message.
   */
  readonly messages: readonly ChatMessage[];

  /**
   * Performance statistics for each generation step in this turn.
   * If the model executed tools, this array contains one entry per generation step.
   */
  readonly stats: readonly LLMGenerationStats[];

  /**
   * The termination reason:
   * - 'stop': The model generated an End-Of-Sequence (EOS) token or hit maxNewTokens.
   * - 'maxToolTurns': The turn was terminated because tool execution reached maxToolTurns.
   */
  readonly finishReason: 'stop' | 'maxToolTurns';
};

stats is an array of LLMGenerationStats because tool calling can trigger multiple consecutive generation steps in a single turn (e.g. stats[0] for the model generating the tool call, and stats[1] for generating the final answer after tool execution). Each entry provides numPromptTokens, numGeneratedTokens, prefillDurationMs, and start/end timestamps (inferenceStartMs / inferenceEndMs) to compute tokens per second (tok/s).

Chat Templates & Incremental KV Cache Diffing​

Under the hood, createChatPreprocessor renders ChatMessage[] arrays using the model's official Jinja2 template from tokenizer_config.json (formatting special tokens, roles, and generation headers).

To keep multi-turn chat responsive without re-encoding past history on every message, the preprocessor uses incremental prompt diffing:

  1. It verifies that the rendered prefix of previously committed turns is an exact substring match of the newly updated conversation.
  2. It slices out only the newly appended tokens and passes them to runner.prefill().
  3. The existing Key-Value (KV) cache in native memory is preserved, so generation starts immediately without recalculating prior turns.
Custom Model Compatibility

If you use a custom model whose Jinja template dynamically rewires earlier turns when new messages arrive (breaking monotonicity), pass resetOnTurn: true in your session options to force full re-encoding each turn.

Multimodal Inputs​

Vision-Language Models (such as Liquid AI's LFM2_5_VL_450M and LFM2_5_VL_1_6B) process interleaved text and visual payloads:

import { models, useLLMChatSession } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function VisionChat() {
  const session = useLLMChatSession(models.llm.LFM2_5_VL_450M.DEFAULT);

  const handleAnalyzePhoto = async (image: ImageBuffer) => {
    if (!session.isReady || !session.sendMessage) return;

    // Send array of interleaved media and text
    const turn = await session.sendMessage(
      [
        { kind: 'image', image },
        'What type of flower is this, and what care instructions should I follow?',
      ],
      (token) => {
        process.stdout.write(token);
      }
    );

    console.log('Result:', turn.messages);
  };
}

The session automatically embeds the model's sentinel vision tokens, resizes and normalizes the image buffer to the vision encoder's target shape, and feeds the resulting image tensors into the multimodal execution runner alongside text tokens.

Automated Tool Calling​

The LLM chat session supports automated, multi-turn tool calling (function calling). When tool definitions and a parser are supplied, the session automatically invokes tool callbacks, feeds their results back into the conversation, and returns the final assistant answer.

1. Define Tools with execute​

Declare tools matching standard JSON Schema specifications along with an asynchronous execute handler:

import { type ToolDefinition } from 'react-native-executorch/llm';

export const weatherTool: ToolDefinition<{ location: string; unit?: 'celsius' | 'fahrenheit' }> = {
  type: 'function',
  function: {
    name: 'get_current_weather',
    description: 'Get the current weather conditions for a given city.',
    parameters: {
      type: 'object',
      properties: {
        location: { type: 'string', description: 'City name' },
        unit: { type: 'string', enum: ['celsius', 'fahrenheit'] },
      },
      required: ['location'],
    },
  },
  execute: async ({ location, unit = 'celsius' }) => {
    // Query local device sensors or web API
    return JSON.stringify({ location, temperature: 22, unit, condition: 'Sunny' });
  },
};

2. Supply a Tool Parser (parseToolCalls)​

Because different open-source model families emit tool calls in varying syntax (e.g. XML <tool_call> tags, JSON markdown blocks, or special tokens), supply a parser function conforming to ToolParser:

import type { ToolParser, ToolParserResult } from 'react-native-executorch/llm';

// Example parser for XML tag syntax:
// <tool_call>{"name": "get_current_weather", "arguments": {"location": "San Francisco"}}</tool_call>
export const xmlToolParser: ToolParser = (text: string): ToolParserResult | undefined => {
  const match = text.match(/<tool_call>([\s\S]*?)<\/tool_call>/);
  if (!match) return undefined;

  try {
    const json = JSON.parse(match[1].trim());
    return {
      toolCalls: [
        {
          function: {
            name: json.name,
            arguments: json.arguments,
          },
        },
      ],
      // Residual text outside the tool call tags
      textContent: text.replace(match[0], '').trim(),
    };
  } catch {
    return undefined;
  }
};

3. Attach to Session​

const session = useLLMChatSession(models.llm.LFM2_5_1_2B.DEFAULT, {
  toolOpts: {
    tools: [weatherTool],
    parseToolCalls: xmlToolParser,
    maxToolTurns: 5, // Maximum consecutive tool execution turns before halting
  },
});

Imperative Session API​

For headless background services or non-React architectures, create a full chat session imperatively using createLLMChatSession:

import { createLLMChatSession, download, models } from 'react-native-executorch';

// Download and cache LLM weights and tokenizer files
const model = await download(models.llm.LFM2_5_1_2B.DEFAULT);
const session = await createLLMChatSession(model, {
  initialMessages: [{ role: 'system', content: 'You are an offline assistant.' }],
  generationConfig: { temperature: 0.3, maxNewTokens: 256 },
});

try {
  const result = await session.sendMessage("Summarize today's logs.");
  console.log('Answer:', result.messages[result.messages.length - 1].content);
} finally {
  session.dispose();
}

Low-Level Runner​

While useLLMChatSession and createLLMChatSession handle chat formatting, message histories, and automated tool calling loops, you can drop down directly to the native LLMRunner via llm.createLLMRunner().

LLMRunner operates synchronously on a worklet runtime thread and provides low-level control:

  • Raw Prompt Ingestion: Pass raw prompt strings or preprocessed image tensors directly to the runner without role formatting or Jinja chat template rendering.
  • Manual Prefill: Execute runner.prefill(prompt) to populate the Key-Value (KV) cache with large background contexts, system prompts, or document chunks before starting interactive generation.
  • Direct Synchronous Generation: Call runner.generate(prompt, config, onToken) to generate text continuations with zero Promise scheduling overhead, executing the onToken callback directly on each generated token.
  • KV Cache Inspection & Slicing: Query runner.getKVCacheState() to check occupied tokens (pos), max context length (maxSeqLen), and context capacity ratio (usageRatio).
  • KV Cache Rewind & Branching: Call runner.reset(targetPos) to rewind the KV cache back to an exact token position. This enables speculative branching, sampling multiple divergent continuations from a shared prompt prefix without re-encoding, or manual conversation tree management.
  • Cancellation: Call runner.stop() from any thread to abort active autoregressive generation immediately.

Available Models​

The library provides ready-to-use models from the Software Mansion HuggingFace LLM Collection, pre-packaged with their tokenizers and Jinja chat templates in models.llm:

Model FamilyVariantsSize RangeSupported BackendsNotes
Liquid LFM 2.5350M, 1.2B, VL 450M, VL 1.6B265 MB – 2.43 GBXNNPACK (CPU), MLX (Apple), Vulkan (Android)Fast hybrid RNN/Transformer for low-latency chat & visual reasoning.
Meta Llama 3.21B, 3B1.06 GB – 5.99 GBXNNPACK (CPU)High-quality reasoning, summarization, and instruction following.
Google Gemma 4E2B2.45 GB – 2.70 GBXNNPACK (CPU), MLX (Apple), Vulkan (Android)High-fidelity instruction following from Google DeepMind research.
Alibaba Qwen 30.6B, 1.7B, 4B482 MB – 7.49 GBXNNPACK (CPU)Next-gen compact multilingual models supporting 29+ languages.
Alibaba Qwen 2.50.5B, 1.5B, 3B417 MB – 5.75 GBXNNPACK (CPU)Proven multilingual instruction models across code, math, and chat.
Hammer 2.10.5B, 1.5B, 3B398 MB – 5.75 GBXNNPACK (CPU)Fine-tuned function calling for automated tool execution & structured JSON.
Microsoft Phi-4 Mini3.8B2.62 GB – 7.15 GBXNNPACK (CPU)High-density reasoning model for STEM problem solving & coding.
SpeakLeash Bielik v31.5B923 MB – 2.97 GBXNNPACK (CPU)Bilingual Polish & English instruction model.
Using Custom Models

To use your own fine-tuned LLM .pte model, pass an LLMModel configuration object to useLLMChatSession or createLLMChatSession:

const customSession = await createLLMChatSession({
  modelPath: 'https://example.com/my-llm.pte',
  tokenizerPath: 'https://example.com/tokenizer.json',
  tokenizerConfigPath: 'https://example.com/tokenizer_config.json',
});

The pipeline automatically verifies that the model's exported methods and KV cache tensors match its requirements. To prepare and export your own .pte model to match this pipeline, see Exporting Custom Models.

API Reference​

Hooks & Pipelines​

Types & Options​

  • LLMChatSession — Active chat session interface (sendMessage, stop, getHistory, getKVCacheState, dispose).
  • LLMRunner — Low-level runner interface (prefill, generate, reset, getKVCacheState).
  • ChatPreprocessor — Chat formatting and diffing preprocessor interface.
  • ToolDefinition — Tool declaration with JSON Schema parameters and execute callback.
  • ToolParser — Parser function type for extracting tool calls from model output.
  • LLMChatTurnResult — Result of a chat turn with updated messages, finish reason, and performance statistics.
  • LLMKVCacheState — KV cache metrics (pos, maxSeqLen, usageRatio).
  • LLMChatSessionOptions — Session configuration options (generationConfig, initialMessages, toolOpts).
  • LLMModel — Model configuration spec with model, tokenizer, and tokenizer config paths.
  • LLMGenerationConfig — Sampling and decoding parameters (temperature, topP, maxNewTokens).
  • ChatMessage — Standard chat message structure (role, content).

Model Presets​

  • models.llm — Pre-configured LLM models registry.
Source Code

View the implementation on GitHub:

We are Software Mansion.

Hire us