espnet2.text.build_tokenizer.build_tokenizer

source

espnet2.text.build_tokenizer.build_tokenizer(token_type: str, bpemodel: Path | str | Iterable[str] | None = None, non_linguistic_symbols: Path | str | Iterable[str] | None = None, remove_non_linguistic_symbols: bool = False, space_symbol: str = '<space>', delimiter: str | None = None, g2p_type: str | None = None, nonsplit_symbol: Iterable[str] | None = None, encode_kwargs: Dict | None = None, whisper_language: str | None = None, whisper_task: str | None = None, sot_asr: bool = False) → AbsTokenizer

A helper function to instantiate Tokenizer