Benchmarks

Benchmarks of ruNNtime against the alternatives. Same model, same input, same browser, one engine at a time: ruNNtime zoo, ruNNtime through the transformers.js plugin, and stock transformers.js on ONNX Runtime, all on WebGPU. All tests were performed on an Apple M2 Pro.

Speech to text

Moonshine on 500 utterances of LibriSpeech test-clean, 66 minutes of speech. ruNNtime runs the f16 weights in half precision, its default. transformers.js runs the fp32 and q4 ONNX exports.

× realtime

Seconds of audio transcribed per second. Higher is better.

• ruNNtime tiny137×
• ruNNtime tiny, tjs plugin139×
• transformers.js tiny fp3220×
• transformers.js tiny q420×
• ruNNtime base78×
• ruNNtime base, tjs plugin78×
• transformers.js base fp3215×
• transformers.js base q415×
• ruNNtime streaming tiny123×

ms per token

Decode time per generated token. Lower is better.

• ruNNtime tiny1.8 ms
• ruNNtime tiny, tjs plugin1.7 ms
• transformers.js tiny fp3212.1 ms
• transformers.js tiny q412.3 ms
• ruNNtime base2.6 ms
• ruNNtime base, tjs plugin2.6 ms
• transformers.js base fp3216.3 ms
• transformers.js base q415.9 ms
• ruNNtime streaming tiny1.9 ms

All numbers

Model	Engine	Precision	× realtime	ms per token
tiny	ruNNtime	f16	137	1.8
tiny	ruNNtime, tjs plugin	f16	139	1.7
tiny	transformers.js	fp32	20	12.1
tiny	transformers.js	q4	20	12.3
base	ruNNtime	f16	78	2.6
base	ruNNtime, tjs plugin	f16	78	2.6
base	transformers.js	fp32	15	16.3
base	transformers.js	q4	15	15.9
streaming tiny	ruNNtime	f16	123	1.9

Text embedding

all-MiniLM-L6-v2 on 1000 Wikipedia paragraphs, 22 to 629 words each, in batches of 1, 8 and 32 paragraphs per call. ruNNtime runs the f16 weights in half precision, its default. transformers.js runs the fp32 ONNX export, its WebGPU default, plus the fp16 and q4 exports.

Paragraphs per second

Higher is better.

Batch of 32

• ruNNtime596/s
• ruNNtime, tjs plugin230/s
• transformers.js fp3287/s
• transformers.js fp16128/s
• transformers.js q493/s

All numbers

Milliseconds per paragraph.

Engine	Precision	Batch 1	Batch 8	Batch 32
ruNNtime	f16	3.4	1.8	1.6
ruNNtime	f32	4.1	2.5	2.3
ruNNtime, tjs plugin	f16	5.5	3.5	4.2
transformers.js	fp32	7.8	7.0	10.8
transformers.js	fp16	6.8	5.0	7.9
transformers.js	q4	6.9	7.0	10.6

Embedding (EmbeddingGemma 2)

EmbeddingGemma 2 on text, images, audio and video. ruNNtime runs the f16 weights in half precision, its default, and the f32 weights. transformers.js runs the fp32 ONNX export, its WebGPU default, plus the fp16, q4 and q4f16 exports. The model is not in the plugin, so there is no plugin row.

Text

1000 paragraphs in batches of 1, 8 and 32 paragraphs per call.

Paragraphs per second, batch of 32

Higher is better.

• ruNNtime40.5/s
• transformers.js fp3210.2/s
• transformers.js q49.2/s

Paragraphs per second, batch of 1

Higher is better.

• ruNNtime27.6/s
• transformers.js fp3217.7/s
• transformers.js fp1623.7/s
• transformers.js q417.1/s
• transformers.js q4f1620.5/s

All numbers

Paragraphs per second.

Engine	Precision	Batch 1	Batch 8	Batch 32
ruNNtime	f16	27.6	39.8	40.5
ruNNtime	f32	20.5	26.0	26.6
transformers.js	fp32	17.7	14.3	10.2
transformers.js	fp16	23.7	21.9	16.2
transformers.js	q4	17.1	12.9	9.2
transformers.js	q4f16	20.5	18.7	13.9

At batch 32, the transformers.js fp16 and q4f16 vectors do not match ruNNtime’s, so those two cells are not in the chart.

Images

The first 100 COCO photos, one photo per call.

Images per second

Higher is better.

• ruNNtime1.73/s
• transformers.js fp321.01/s
• transformers.js fp161.52/s
• transformers.js q40.95/s
• transformers.js q4f161.35/s

All numbers

Engine	Precision	ms per image	Images per second
ruNNtime	f16	576	1.73
ruNNtime	f32	789	1.27
transformers.js	fp32	985	1.01
transformers.js	fp16	656	1.52
transformers.js	q4	1044	0.95
transformers.js	q4f16	739	1.35

Audio

500 LibriSpeech clips, 66 minutes of speech, one clip per call.

× realtime

Seconds of audio embedded per second. Higher is better.

• ruNNtime69.5×
• transformers.js fp3242.9×
• transformers.js fp1661.9×
• transformers.js q442.3×
• transformers.js q4f1652.8×

All numbers

Engine	Precision	ms per clip	Clips per second	× realtime
ruNNtime	f16	94.9	8.81	69.5
ruNNtime	f32	134.7	6.43	50.8
transformers.js	fp32	159.3	5.44	42.9
transformers.js	fp16	109.0	7.84	61.9
transformers.js	q4	157.4	5.36	42.3
transformers.js	q4f16	131.0	6.68	52.8

Video

40 Wikimedia Commons clips, 4 to 14 s each, one frame per second, one clip per call.

Clips per second

Higher is better.

• ruNNtime0.45/s
• transformers.js fp320.23/s
• transformers.js fp160.34/s
• transformers.js q40.20/s
• transformers.js q4f160.31/s

All numbers

Engine	Precision	ms per clip	Clips per second
ruNNtime	f16	2286	0.45
ruNNtime	f32	3335	0.30
transformers.js	fp32	4261	0.23
transformers.js	fp16	3062	0.34
transformers.js	q4	5119	0.20
transformers.js	q4f16	3329	0.31

Image classification

MobileNetV4 on 500 COCO photos, one photo per call, pixels in to class names out. ruNNtime runs the f16 weights, its only precision. transformers.js runs the fp32 ONNX export, its WebGPU default, plus the fp16 and q4 exports. Through the plugin, ruNNtime keeps transformers.js’s image preprocessing, which is most of that row’s time.

FPS

Higher is better.

• ruNNtime407 fps
• ruNNtime, tjs plugin109 fps
• transformers.js fp3295 fps
• transformers.js fp1696 fps
• transformers.js q496 fps

All numbers

Engine	Precision	ms per image	FPS
ruNNtime	f16	2.4	407
ruNNtime, tjs plugin	f16	9.0	109
transformers.js	fp32	10.1	95
transformers.js	fp16	9.9	96
transformers.js	q4	10.0	96

Depth estimation

DepthART on 500 COCO photos, one photo per call, pixels in to a depth map out. ruNNtime runs the f16 weights, its only precision. Through the plugin, ruNNtime runs DepthART with transformers.js’s image preprocessing, which is most of that row’s time.

FPS

Higher is better.

• ruNNtime74.0 fps
• ruNNtime, tjs plugin23.0 fps

All numbers

Engine	Precision	ms per image	FPS
ruNNtime	f16	13.3	74
ruNNtime, tjs plugin	f16	41.4	23

Object detection

YOLO26 on 500 COCO photos, one photo per call, pixels in to boxes out, in the n, s and m sizes. ruNNtime runs the f16 weights, its only precision. transformers.js runs the ONNX export of the same model in fp32, its WebGPU default, plus the fp16 and q4 exports.

FPS

Higher is better.

• ruNNtime n77.1 fps
• transformers.js n fp3224.7 fps
• transformers.js n fp1625.9 fps
• transformers.js n q424.3 fps
• ruNNtime s43.6 fps
• transformers.js s fp3219.5 fps
• transformers.js s fp1621.1 fps
• transformers.js s q419.9 fps
• ruNNtime m25.1 fps
• transformers.js m fp3211.8 fps
• transformers.js m fp1614.3 fps
• transformers.js m q411.9 fps

All numbers

Model	Engine	Precision	ms per image	FPS
n	ruNNtime	f16	12.9	77.1
n	transformers.js	fp32	40.9	24.7
n	transformers.js	fp16	38.1	25.9
n	transformers.js	q4	42.1	24.3
s	ruNNtime	f16	22.7	43.6
s	transformers.js	fp32	48.9	19.5
s	transformers.js	fp16	43.4	21.1
s	transformers.js	q4	49.1	19.9
m	ruNNtime	f16	39.8	25.1
m	transformers.js	fp32	84.9	11.8
m	transformers.js	fp16	69.5	14.3
m	transformers.js	q4	83.7	11.9

The transformers.js rows run the same model and the same compute, but their boxes are wrong: the ONNX export’s config makes transformers.js score classes with softmax instead of sigmoid, so it keeps about 96 boxes per image where ruNNtime finds 4.5.

Pose and instance segmentation

YOLO26 pose and YOLO26 segment on the same 500 COCO photos, one photo per call, in the n, s and m sizes. ruNNtime runs the f16 weights, its only precision. transformers.js has no pipeline for these models, so there is no row to compare with.

Pose, FPS

Higher is better.

• ruNNtime n72.0 fps
• ruNNtime s44.0 fps
• ruNNtime m24.6 fps

Instance segmentation, FPS

Higher is better.

• ruNNtime n60.3 fps
• ruNNtime s34.1 fps
• ruNNtime m16.4 fps

All numbers

Pose:

Model	Engine	Precision	ms per image	FPS
n	ruNNtime	f16	13.6	72.0
s	ruNNtime	f16	22.7	44.0
m	ruNNtime	f16	40.6	24.6

Instance segmentation:

Model	Engine	Precision	ms per image	FPS
n	ruNNtime	f16	16.4	60.3
s	ruNNtime	f16	29.1	34.1
m	ruNNtime	f16	60.9	16.4

Privacy filter

Privacy filter on 300 short messages with personal data, 9 to 76 words each, one message per call, text in to labelled spans out. ruNNtime runs the int8 weights, its only precision. transformers.js runs the ONNX exports of the same model in q4 and q4f16. Its q8 export finds almost none of the spans on WebGPU, so it is left out. Through the plugin, ruNNtime keeps transformers.js’s tokenizer.

Texts per second

Higher is better.

• ruNNtime61.4/s
• ruNNtime, tjs plugin62.1/s
• transformers.js q413.3/s
• transformers.js q4f1614.0/s

All numbers

Engine	Precision	ms per text	Texts per second
ruNNtime	int8	16.4	61.4
ruNNtime, tjs plugin	int8	16.2	62.1
transformers.js	q4	76.4	13.3
transformers.js	q4f16	72.0	14.0

How we measure

• Machine: MacBook Pro, Apple M2 Pro, 16 GB, headless Chromium 156, WebGPU on Metal. Measured October 27 and 28, 2026.
• Setup: ruNNtime as in Getting started, with subgroups and shader-f16, the hosted weights. transformers.js 4.3.1, device: 'webgpu', the onnx-community, Xenova and openai ONNX exports at the row’s dtype.
• Timing: one engine loaded at a time, every input runs once untimed first. Reported numbers are medians.
• Speech to text: the first 500 utterances of LibriSpeech test-clean, 65.8 minutes, 1.4 to 31.6 s per clip, 16 kHz mono, the same samples for every engine. Each clip is timed once, samples in to text out. × realtime is audio seconds over seconds taken, median over the 500 clips. ms per token is decode time over tokens produced, median over clips. Every transcript is checked against the reference.
• Text embedding: the 1000 distinct paragraphs of SQuAD v1.1 dev, Wikipedia text, capped at 512 tokens, the same for every engine. Two passes per batch size, text in to unit-length vectors out. Paragraphs per second is the whole set over the time it took. ms per paragraph is the median call time over the batch size. Every vector is checked against the ruNNtime f32 output.
• EmbeddingGemma 2: text as above, the first 100 COCO val2017 photos, the 500 LibriSpeech clips from speech to text, and 40 Wikimedia Commons clips of 4 to 14 s sampled at one frame per second, the same inputs for every engine. One input per call for images, audio and video. Every vector is checked against ruNNtime’s.
• Vision tasks: image classification, depth estimation, object detection, pose and instance segmentation run on the first 500 COCO val2017 photos under a license that allows reuse, 69k to 410k pixels each, decoded once, the same for every engine. Four passes, RGBA pixels in, the task’s result out at each engine’s default threshold. FPS is the whole set over the time it took. ms per image is the median call time. Every result is checked against ruNNtime’s.
• Privacy filter: 300 English messages, 9 to 76 words, with names, emails, phones, addresses, dates, URLs, account numbers and secrets, the same for every engine. Four passes, text in to labelled spans out. Texts per second is the whole set over the time it took. ms per text is the median call time. Every span set is checked against ruNNtime’s.