Pose & Keypoints

Pose estimation and keypoint detection locate specific anatomical landmarks on detected subjects — such as human skeletal joints (eyes, shoulders, elbows, wrists, hips, knees, ankles) or facial landmarks (eyes, nose tip, mouth, ears). Each prediction outputs a subject bounding box, detection confidence, and landmark coordinates scaled to the input image with individual landmark confidence scores.

Unlike basic object detection (which only returns box boundaries), keypoint detection tracks body posture, movement, and facial alignment. Common use cases include fitness/workout tracking, gesture controls, motion analysis, face alignment, and AR filters.

iOS	Android

Quick Start

The useKeypointDetector hook manages model downloading, initialization, and lifecycle:

import { models, useKeypointDetector } from 'react-native-executorch';
import type { ImageBuffer } from 'react-native-executorch/cv';

function MyComponent() {
  const detector = useKeypointDetector(models.keypointDetection.YOLO26_POSE.DEFAULT);

  // Hook state:
  // detector.isReady          — true once model is downloaded and loaded in memory
  // detector.downloadProgress — 0 to 100 download progress
  // detector.error            — Error instance if download or load failed
  // detector.resource         — resolved config with all URLs replaced by local file paths

  const handleDetect = async (imageBuffer: ImageBuffer) => {
    if (!detector.isReady || !detector.detectKeypoints) return;

    // Run inference on background thread
    const detections = await detector.detectKeypoints(imageBuffer, {
      confidenceThreshold: 0.25,
      iouThreshold: 0.7,
    });
    console.log('Detected poses:', detections);
  };

  // Trigger handleDetect from an image picker, button press, or camera frame
}

Full Interactive Example in Gallery App

See src/app/(screens)/keypoint-detection.tsx in the React Native ExecuTorch Gallery for a complete, runnable screen with photo picker, skeleton keypoint overlays, and latency tracking.

Output Format

detectKeypoints() returns an array of KeypointDetection objects:

type KeypointDetection<F extends BoxFormat = 'xyxy', L extends PropertyKey = string> = {
  /** Scaled bounding box coordinates matching the input image resolution */
  readonly box: BoundingBox<F>;
  /** Overall detection confidence score (between 0.0 and 1.0) */
  readonly confidence: number;
  /** Map of landmark names to their pixel coordinates and confidence scores */
  readonly landmarks: Record<L, { x: number; y: number; confidence: number }>;
};

For human pose models (YOLO26_POSE), landmarks includes 17 COCO_LANDMARKS body points:

[
  {
    box: { format: 'xyxy', xmin: 45.2, ymin: 12.0, xmax: 310.5, ymax: 580.0 },
    confidence: 0.93,
    landmarks: {
      nose: { x: 178.4, y: 85.2, confidence: 0.97 },
      leftEye: { x: 190.1, y: 75.4, confidence: 0.95 },
      rightEye: { x: 165.8, y: 76.0, confidence: 0.94 },
      leftEar: { x: 205.3, y: 80.1, confidence: 0.91 },
      rightEar: { x: 150.2, y: 81.0, confidence: 0.9 },
      // ... 12 more COCO landmarks (shoulders → ankles)
    },
  },
];

For face models (BLAZEFACE), landmarks includes 6 facial points from BLAZEFACE_LANDMARKS: leftEye, rightEye, noseTip, mouthCenter, leftEar, rightEar.

Configuration & Options

Pass a DetectKeypointsOptions object to detectKeypoints() to override model defaults:

Option	Type	Default	Description
confidenceThreshold	number	Model default (e.g. 0.25)	Minimum confidence score for a detected subject to be retained.
iouThreshold	number	Model default (e.g. 0.7)	Non-Maximum Suppression (NMS) IoU overlap threshold.

Imperative API

For background tasks, headless services, or manual lifecycle management outside React components, create the detector using createKeypointDetector:

import { createKeypointDetector, download, models } from 'react-native-executorch';

// Download and cache model assets before creating the pipeline
const model = await download(models.keypointDetection.YOLO26_POSE.DEFAULT);
const detector = await createKeypointDetector(model);

try {
  const poses = await detector.detectKeypoints(imageBuffer, {
    confidenceThreshold: 0.3,
  });
  console.log('Detected poses:', poses);
} finally {
  // Always release native resources when finished
  detector.dispose();
}

Synchronous Execution

For real-time camera tracking and live fitness apps, createKeypointDetector exposes a synchronous detectKeypointsWorklet function. This runs directly on the worklet thread with zero Promise scheduling overhead:

// Called synchronously inside a VisionCamera frame processor on the UI worklet thread
const poses = detector.detectKeypointsWorklet(frameBuffer, {
  confidenceThreshold: 0.3,
});

See Worklets & Threading for details on worklet execution contexts and zero-copy host objects.

Available Models

The library provides ready-to-use pose and landmark detectors from the Software Mansion HuggingFace Pose Estimation Collection, available in models.keypointDetection:

Model Family	Variants	Keypoints Detected	Size Range	Supported Backends	Notes
MediaPipe BlazeFace	See	BLAZEFACE_LANDMARKS (6 facial landmarks + box)	0.6 MB	XNNPACK (CPU)	Ultra-lightweight face bounding box & eye/ear/nose/mouth keypoint tracking (sub-millisecond).
YOLO26 Pose	See	COCO_LANDMARKS (17 body keypoints)	11.4 MB	XNNPACK (CPU), Core ML (Apple)	Real-time multi-person full-body skeletal tracking across multiple input resolutions.
RF-DETR Keypoint	See	COCO_LANDMARKS (17 body keypoints)	138.6 MB – 140.9 MB	XNNPACK (CPU), Core ML (Apple), MLX (Apple)	High-accuracy body keypoint detection transformer for complex, occluded poses.

Using Custom Models

To use your own fine-tuned pose or landmark detection .pte model, pass a KeypointDetectorModel configuration object to useKeypointDetector or createKeypointDetector:

const customDetector = await createKeypointDetector({
  modelPath: 'https://example.com/my-pose-model.pte',
  modelOpts: {
    landmarks: ['head', 'leftHand', 'rightHand'],
    boxFormat: 'xyxy',
    resizeMode: 'letterbox',
    interpolation: 'linear',
    normalizeOpts: { alpha: 1 / 255.0, beta: 0.0 },
    defaultConfidenceThreshold: 0.3,
    defaultIouThreshold: 0.6,
  },
});

The pipeline automatically verifies that the model's exported input and output shapes match its requirements. To prepare and export your own .pte model to match this pipeline, see Exporting Custom Models.

API Reference

Hooks & Pipelines

• useKeypointDetector() — React hook for keypoint detector downloading, state, and lifecycle.
• createKeypointDetector() — Imperative factory for keypoint and pose detection pipelines.

Types & Options

• KeypointDetector — Keypoint detector runner interface (detectKeypoints, detectKeypointsWorklet).
• KeypointDetection — Detection result structure containing box, confidence, and landmarks.
• DetectKeypointsOptions — Detection options (confidenceThreshold, iouThreshold).
• KeypointDetectorModel — Model configuration spec for pose and landmark models.
• KeypointDetectorOptions — Options defining landmark names, box format, and normalization.
• Landmarks — Record of landmark names mapped to { x, y, confidence }.
• BoundingBox — Bounding box structure.
• ImageBuffer — Input image buffer structure.

Model Presets & Constants

• models.keypointDetection — Pre-configured keypoint and pose models registry.
• COCO_LANDMARKS — List of 17 standard COCO skeletal body keypoints.
• BLAZEFACE_LANDMARKS — List of 6 standard BlazeFace facial landmarks.

Source Code

View the implementation on GitHub:

• src/extensions/cv/tasks/keypointDetection.ts ↗