espnet3.systems.f5tts.src.create_token_list.create_token_list

source

espnet3.systems.f5tts.src.create_token_list.create_token_list(config: DictConfig) → None

Write the token list of the training transcripts to a file.

The body of the create_token_list stage. It reads the transcript column of the training manifest, cleans and tokenizes it with ESPnet2’s text front end, and writes one token per line to save_path/filename, most frequent first with the configured special symbols inserted.

Configuration should include (under create_token_list):

• save_path: Directory in which to save the token-list file.
• filename: Token-list file name, such as tokens.txt.
• manifest_path: Training manifest. Defaults to data/manifest/train.tsv, relative to the working directory.
• token_type: char, word, bpe or phn. Defaults to char.
• cleaner: Optional text-cleaner name, such as tacotron.
• g2p: Optional grapheme-to-phoneme model name.
• bpemodel, delimiter, space_symbol, non_linguistic_symbols, remove_non_linguistic_symbols: Forwarded to espnet2.text.build_tokenizer.build_tokenizer.
• add_symbol / add_nonsplit_symbol: Special symbols to insert, each written "<symbol>:<index>"; a negative index counts from the end.
• cutoff: Drop tokens seen this many times or fewer. Defaults to 0, which keeps every token.
• vocabulary_size: Upper bound on the list length, special symbols included. 0 or a negative value means unlimited.
• vocab_builder / vocab_builder_conf: Dotted path of a callable builder(texts: list[str], **vocab_builder_conf) -> list[str] and its keyword arguments. When set, the callable receives the cleaned transcripts and returns the full ordered token list, replacing the frequency-count construction and the special-symbol handling above.

• Parameters:config – Training config holding the create_token_list block.
• Raises:RuntimeError – If the create_token_list block, save_path or filename is missing, the manifest does not exist, a manifest row has fewer than three tab-separated columns, vocabulary_size is smaller than the number of add_symbol entries, or a special symbol is not written "<symbol>:<index>".

Example

create_token_list:
  save_path: ${data_dir}/token_list
  filename: tokens.txt
  manifest_path: ${data_dir}/manifest_filtered/train.tsv
  token_type: char
  add_symbol:
    - "<blank>:0"
    - "<unk>:1"
    - "<sos/eos>:-1"

With a custom vocabulary builder, here F5-TTS’s zh+en pinyin one:

create_token_list:
  save_path: ${data_dir}/token_list
  filename: vocab.txt
  vocab_builder: espnet3.systems.f5tts.src.pinyin.build_pinyin_vocab

With the first config and a training manifest whose only transcript is hello world:

>>> create_token_list(training_config)
>>> Path("data/token_list/tokens.txt").read_text().splitlines()
['<blank>', '<unk>', 'l', 'o', 'h', 'e', '<space>', 'w', 'r', 'd',
 '<sos/eos>']

NOTE

Running the stage again overwrites the token list. Blank lines and rows whose transcript column is empty contribute no tokens. A row with no transcript column is reported as an error with its line number instead of being skipped, so a malformed manifest is noticed.