Function: useTokenizer()

useTokenizer(tokenizerPath, options?): object

Defined in: hooks/useTokenizer.ts:21

React hook to load and run a tokenizer.

This hook manages downloading (if remote URLs are provided) and loading the tokenizer configuration files, tracking download progress and load errors, and releasing native memory when the component unmounts or the configuration changes.

For imperative usage, see createTokenizer.

Parameters

tokenizerPath

string

A remote URL or local path to a tokenizer.json file.

options?

ResourceOptions

Load and caching options. See ResourceOptions.

Returns

object

The same object as nlp.Tokenizer (without dispose), combined with loading state and download progress.

decode

decode: any = model.decode

downloadProgress

downloadProgress: any

encode

encode: any = model.encode

error

error: any

getVocabSize

getVocabSize: any = model.getVocabSize

idToToken

idToToken: any = model.idToToken

isReady

isReady: boolean = !!model

resource

resource: any

tokenToId

tokenToId: any = model.tokenToId

See

nlp.Tokenizer