espnet3.systems.esp2_st.system.ESP2STSystem
espnet3.systems.esp2_st.system.ESP2STSystem
class espnet3.systems.esp2_st.system.ESP2STSystem(training_config: DictConfig | None = None, inference_config: DictConfig | None = None, metrics_config: DictConfig | None = None, publication_config: DictConfig | None = None, demo_config: DictConfig | None = None)
Bases: BaseSystem
System for speech translation, with one vocabulary per side.
espnet2.tasks.st.STTask carries two vocabularies, not one:
- the target side feeds the ST decoder (
criterion_st,size=vocab_size) and the ST-CTC; - the source side feeds the auxiliary ASR decoder (
criterion_asr,size=src_vocab_size) and the ASR-CTC.
They are independent – different vocabulary sizes, and in egs2 different casing conventions too (tgt_case=tc truecased, src_case=lc.rm lowercased and stripped of punctuation). ASRSystem trains a single model and so cannot express this; this subclass trains one per side.
Config shape under tokenizer:
tokenizer:
tgt:
vocab_size: 4000
model_type: bpe
save_path: ${data_dir}/bpe_tgt_4000
text_builder: {_target_: ..., ...}
src:
vocab_size: 4000
model_type: bpe
save_path: ${data_dir}/bpe_src_4000
text_builder: {_target_: ..., ...}The model config then points at the two token lists:
model:
token_list: ${tokenizer.tgt.save_path}/tokens.txt
src_token_list: ${tokenizer.src.save_path}/tokens.txtA side may be omitted, in which case it is skipped – egs2’s use_src_lang=false trains the target model only.
Examples
>>> [s.name for s in ESP2STSystem.stages[:2]]
['create_dataset', 'train_tokenizer']Initialize the system with optional stage configs.
- Parameters:
- training_config – Training configuration for data preparation, statistics collection, and model training.
- inference_config – Inference configuration used by the
inferstage. - metrics_config – Measurement configuration used by the
measurestage. - publication_config – Publication configuration for
pack_modelandupload_modelstages. - demo_config – Demo configuration for the
demostage.
collect_stats(*args, **kwargs)
Collect stats, then make the token shape files 2-D.
espnet2’s st.sh bounds batch_bins over THREE shape files – speech_shape plus one per text stream – and writes the text ones as "L,V" by appending the vocabulary size, so one token costs V bins. Without that the batcher cannot see text length at all and a batch of short utterances grows until it exhausts GPU memory.
collect_stats records the token streams as 1-D lengths; only the System knows which vocabulary belongs to which stream, so the vocabulary size is appended here. text takes the target vocabulary and src_text the source one, matching STTask.
stages : ClassVar[tuple[StageSpec, ...]] = (StageSpec(name='create_dataset', config='training', log_dir='data_dir'), StageSpec(name='train_tokenizer', config='training', log_dir='tokenizer.tgt.save_path'), StageSpec(name='collect_stats', config='training', log_dir='stats_dir'), StageSpec(name='train', config='training', log_dir='exp_dir'), StageSpec(name='infer', config='inference', log_dir='inference_dir'), StageSpec(name='measure', config='metrics', log_dir='inference_dir'), StageSpec(name='pack_model', config=('training', 'publication'), log_dir=None), StageSpec(name='upload_model', config='publication', log_dir=None), StageSpec(name='pack_demo', config='demo', log_dir='pack.out_dir'), StageSpec(name='upload_demo', config='demo', log_dir='pack.out_dir'))
train(*args, **kwargs)
Train the model, training any missing tokenizer first.
- Raises:RuntimeError – If neither
datasetnordataset_diris set on the training config.
train_tokenizer(*args, **kwargs)
Train the target and source SentencePiece models.
Expected config – one block per side, each side optional except tgt:
tokenizer:
tgt:
vocab_size: 4000
model_type: bpe
save_path: ${data_dir}/bpe_tgt_4000
text_builder:
_target_: egs3.must_c.esp2_st.dataset.gather_training_text
recipe_dir: ${recipe_dir}
tgt_lang: de
src:
vocab_size: 4000
model_type: bpe
save_path: ${data_dir}/bpe_src_4000
text_builder: {...}text_builder is a Hydra config: _target_ names a callable that returns a path or an iterable of lines, and every other key is passed to it as a keyword argument. Set train_file to read prepared text instead, and reuse_existing_text: false to make a leftover file an error rather than silently reused.
What it writes, per side:
data/bpe_tgt_4000/bpe.model SentencePiece model
data/bpe_tgt_4000/bpe.vocab SentencePiece vocabulary
data/bpe_tgt_4000/tokens.txt one token per line, the token_list
data/train_tokenizer/tgt.txt the gathered training texttokens.txt is what the model config points token_list and src_token_list at, and its line count is the vocabulary size collect_stats appends to the text shapes.
A side whose model and tokens.txt already exist is skipped, so a rerun after adding the source side trains only the missing one.
- Raises:RuntimeError – If no side is configured, or a side’s builder produces no text.
