espnet3.systems.f5tts.src.pinyin.build_pinyin_vocab

source

espnet3.systems.f5tts.src.pinyin.build_pinyin_vocab(texts: Iterable[str], add_ascii_latin: bool = False, polyphone: bool = True) → List[str]

Build an F5-style pinyin vocab.txt token list from raw transcripts.

Reproduces the vocab construction in F5-TTS’s dataset prep (prepare_emilia.py / prepare_csv_wavs.py): tokenize every transcript with convert_char_to_pinyin(), take the union of unique tokens, and return them sorted (Unicode codepoint order). No <blank>/<unk>/ <sos/eos> symbols are added, and a literal space lands at index 0 (it is the lowest-codepoint token), matching F5’s convention (vocab[" "] == 0).

texts is any iterable of strings, so this is dataset-agnostic. Feed it transcripts from any manifest or dataset.

• Parameters:
  • texts – Iterable of raw transcripts (zh/en/mixed).
  • add_ascii_latin – If True, also seed the vocab with ASCII (32..126) and Latin-1 (192..255) characters, F5’s optional line for broadening coverage (e.g. finetuning to de/fr). Off by default to match the plain Emilia/csv prep.
  • polyphone – Passed through to convert_char_to_pinyin().
• Returns: Sorted list of vocab tokens (write one per line to vocab.txt).

NOTE

The exact token set depends on the corpus and on the rjieba / pypinyin versions (polyphone handling), so this will not necessarily be byte-identical to F5’s published Emilia_ZH_EN_pinyin/vocab.txt. For from-scratch training that is fine; for finetuning a pretrained F5 model, use F5’s shipped vocab.txt instead.

Example

>>> build_pinyin_vocab(["ab"])
['a', 'b']