espnet3.components.data.data_organizer.DataOrganizer

About 3 min

espnet3.components.data.data_organizer.DataOrganizer

Bases: object

Organizes training, validation, and test datasets into a unified interface.

This class constructs combined datasets for training and validation, and individual named datasets for testing, optionally applying a transform and preprocessor per dataset.

Parameters:
- train (Optional *[*List *[*Union [DatasetConfig , Dict *[*str , Any ] , DictConfig ] ] ]) – A list of training dataset configuration objects.
- valid (Optional *[*List *[*Union [DatasetConfig , Dict *[*str , Any ] , DictConfig ] ] ]) – A list of validation dataset configuration objects.
- test (Optional *[*List *[*Union [DatasetConfig , Dict *[*str , Any ] , DictConfig ] ] ]) – A list of test dataset configurations, each with a name and corresponding data source and optional transform.
- preprocessor (Optional *[*Callable ]) –
  A global preprocessor function or Hydra config applied after each dataset’s transform. If it is an instance of AbsPreprocessor, each sample is passed as (uid, sample). Dict-style configs support both shared and split-specific forms:
  - Shared config: the same preprocessor is used for train/valid/test.
  - Split config: train, valid, and test can each define their own _target_ block.
  - Shared keys outside train/valid/test are merged into each split-specific config. This is useful for common tokenizer settings such as token_type, token_list, and bpemodel.
  Split config fallback rules are:
  - If test is missing, test uses the valid preprocessor.
  - If valid is missing, valid uses the train preprocessor.
  - If train is missing, train uses no preprocessor.
- recipe_dir (Optional *[*str ]) – Recipe root used to resolve local dataset modules when dataset entries omit data_src and rely on recipe_dir/dataset.
- train_mode (bool) – Build the train split’s ESPnet preprocessor in training mode (train=True: data augmentation and the other train-only steps). collect_stats passes False by default, as ESPnet2 collects statistics with build_preprocess_fn(args, train=False): random augmentation would make the statistics and shape files differ from run to run (a training config can turn it back on with collect_stats_train_mode: true). Valid and test are never in training mode.

train

Combined dataset built from training configurations, or None if not provided.

Type:CombinedDataset

valid

Combined dataset built from validation configurations, or None if not provided.

Type:CombinedDataset

test_sets

Dictionary mapping test set names to DatasetWithTransform instances.

Type: Dict[str, DatasetWithTransform]
Raises:
- RuntimeError – If only one of train or valid is provided.
- RuntimeError – If train and valid are of mismatched types (e.g., one is CombinedDataset, the other is None).
- AssertionError – If preprocessor is not callable.

Notes

The DataOrganizer is designed to support both training and testing workflows:

For training: provide both train and valid.
For testing only: provide test and omit train / valid.
All three (train, valid, test) can also be provided simultaneously.

If any of the train, valid, or test are omitted, the corresponding : attributes will be set to None or empty.

Split-specific preprocessor configs are useful when train and valid should not share augmentation behavior.

Examples

Training and validation with one shared preprocessor:

organizer = DataOrganizer( … train=training_configs, … valid=valid_configs, … preprocessor=MyPreprocessor() … ) sample = organizer.train[0]

Testing only:

organizer = DataOrganizer( … test=test_configs, … preprocessor=MyPreprocessor() … ) test_sample = organizer.test[“test_clean”][0]

Shared Hydra config for all splits: : ```python

config = { ... "target": "my_project.preprocess.BuildPreprocessor", ... "token_type": "bpe", ... "token_list": "tokens.txt", ... } organizer = DataOrganizer( ... train=training_configs, ... valid=valid_configs, ... preprocessor=config, ... )


Split-specific Hydra config with shared tokenizer settings:
: ```python
>>> config = {
...     "token_type": "bpe",
...     "token_list": "tokens.txt",
...     "bpemodel": "bpe.model",
...     "train": {
...         "_target_": "my_project.preprocess.TrainPreprocessor",
...         "speed_perturb_prob": 0.1,
...     },
...     "valid": {
...         "_target_": "my_project.preprocess.ValidPreprocessor",
...     },
... }
>>> organizer = DataOrganizer(
...     train=training_configs,
...     valid=valid_configs,
...     test=test_configs,
...     preprocessor=config,
... )

Split-specific config with test fallback to valid: : ```python

config = { ... "train": { ... "target": "my_project.preprocess.TrainPreprocessor", ... }, ... "valid": { ... "target": "my_project.preprocess.ValidPreprocessor", ... }, ... } organizer = DataOrganizer( ... train=training_configs, ... valid=valid_configs, ... test=test_configs, ... preprocessor=config, ... )


The train split without augmentation, as the `collect_stats` stage
builds it:

```python
>>> organizer = DataOrganizer(
...     train=training_configs,
...     valid=valid_configs,
...     preprocessor=config,
...     train_mode=False,
... )

A recipe does not set train_mode here: the same dataset: block of the training config serves both stages, and its augmentation applies in train but not in collect_stats. The training config’s top-level collect_stats_train_mode: true applies it in collect_stats too:

dataset:
  _target_: espnet3.components.data.data_organizer.DataOrganizer
  train:
    - data_src_args:
        split: train
  valid:
    - data_src_args:
        split: valid
  preprocessor:
    _target_: espnet2.train.preprocessor.CommonPreprocessor
    data_aug_effects:
      - [0.1, "contrast", {"enhancement_amount": 75.0}]
    data_aug_num: [1, 1]
    data_aug_prob: 1.0
    token_type: bpe
    token_list: ${tokenizer.save_path}/tokens.txt
    bpemodel: ${tokenizer.save_path}/bpe.model

Initialize DataOrganizer object.

log_summary(log: Logger | None = None) → None

Log a concise dataset summary.

property test

Get the dictionary of test datasets.