espnet2.tasks.abs_task.AbsTask

source

class espnet2.tasks.abs_task.AbsTask

Bases: ABC

abstract classmethod add_task_arguments(parser: ArgumentParser)

classmethod build_category_chunk_iter_factory(args: Namespace, iter_options: IteratorOptions, mode: str) → AbsIterFactory

classmethod build_category_iter_factory(args: Namespace, iter_options: IteratorOptions, mode: str) → AbsIterFactory

classmethod build_chunk_iter_factory(args: Namespace, iter_options: IteratorOptions, mode: str) → AbsIterFactory

abstract classmethod build_collate_fn(args: Namespace, train: bool) → Callable[[Sequence[Dict[str, ndarray]]], Dict[str, Tensor]]

Return “collate_fn”, which is a callable object and given to DataLoader.

>>> from torch.utils.data import DataLoader
>>> loader = DataLoader(collate_fn=cls.build_collate_fn(args, train=True), ...)

In many cases, you can use our common collate_fn.

classmethod build_iter_factory(args: Namespace, distributed_option: DistributedOption, mode: str, kwargs: dict | None = None) → AbsIterFactory

Build a factory object of mini-batch iterator.

This object is invoked at every epochs to build the iterator for each epoch as following:

>>> iter_factory = cls.build_iter_factory(...)
>>> for epoch in range(1, max_epoch):
...     for keys, batch in iter_fatory.build_iter(epoch):
...         model(**batch)

The mini-batches for each epochs are fully controlled by this class. Note that the random seed used for shuffling is decided as “seed + epoch” and the generated mini-batches can be reproduces when resuming.

Note that the definition of “epoch” doesn’t always indicate to run out of the whole training corpus. “–num_iters_per_epoch” option restricts the number of iterations for each epoch and the rest of samples for the originally epoch are left for the next epoch. e.g. If The number of mini-batches equals to 4, the following two are same:

• 1 epoch without “–num_iters_per_epoch”
• 4 epoch with “–num_iters_per_epoch” == 1

classmethod build_iter_options(args: Namespace, distributed_option: DistributedOption, mode: str)

abstract classmethod build_model(args: Namespace) → AbsESPnetModel

classmethod build_model_from_file(config_file: Path | str | None = None, model_file: Path | str | None = None, device: str = 'cpu') → Tuple[AbsESPnetModel, Namespace]

Build model from the files.

This method is used for inference or fine-tuning.

• Parameters:
  • config_file – The yaml file saved when training.
  • model_file – The model file saved when training.
  • device – Device type, “cpu”, “cuda”, or “cuda:N”.

classmethod build_multiple_iter_factory(args: Namespace, distributed_option: DistributedOption, mode: str)

classmethod build_optimizers(args: Namespace, model: Module) → List[Optimizer]

abstract classmethod build_preprocess_fn(args: Namespace, train: bool) → Callable[[str, Dict[str, array]], Dict[str, ndarray]] | None

classmethod build_sequence_iter_factory(args: Namespace, iter_options: IteratorOptions, mode: str) → AbsIterFactory

classmethod build_streaming_iterator(data_path_and_name_and_type, preprocess_fn, collate_fn, key_file: str | None = None, batch_size: int = 1, dtype: str = <class 'numpy.float32'>, num_workers: int = 1, allow_variable_data_keys: bool = False, ngpu: int = 0, inference: bool = False, mode: str | None = None, multi_task_dataset: bool = False) → DataLoader

Build DataLoader using iterable dataset

classmethod build_task_iter_factory(args: Namespace, iter_options: IteratorOptions, mode: str) → AbsIterFactory

Build task specific iterator factory

Example

>>> class YourTask(AbsTask):
... @classmethod
... def add_task_arguments(cls, parser: argparse.ArgumentParser):
...     parser.set_defaults(iterator_type="task")
...
... @classmethod
... def build_task_iter_factory(
...     cls,
...     args: argparse.Namespace,
...     iter_options: IteratorOptions,
...     mode: str,
... ):
...     return FooIterFactory(...)
...
... @classmethod
... def build_iter_options(
....    args: argparse.Namespace,
...     distributed_option: DistributedOption,
...     mode: str
... ):
...     # if you need to customize options object

classmethod check_required_command_args(args: Namespace)

classmethod check_task_requirements(dataset: AbsDataset | IterableESPnetDataset, allow_variable_data_keys: bool, train: bool, inference: bool = False) → None

Check if the dataset satisfy the requirement of current Task

class_choices_list : List[[ClassChoices](../train/ClassChoices.md#espnet2.train.class_choices.ClassChoices)] = []

classmethod exclude_opts() → Tuple[str, ...]

The options not to be shown by –print_config

classmethod get_default_config() → Dict[str, Any]

Return the configuration as dict.

This method is used by print_config()

classmethod get_parser() → ArgumentParser

classmethod main(args: Namespace | None = None, cmd: Sequence[str] | None = None)

classmethod main_worker(args: Namespace)

num_optimizers : int = 1

abstract classmethod optional_data_names(train: bool = True, inference: bool = False) → Tuple[str, ...]

Define the optional names by Task

This function is used by

cls.check_task_requirements() If your model is defined as follows,

>>> from espnet2.train.abs_espnet_model import AbsESPnetModel
>>> class Model(AbsESPnetModel):
...     def forward(self, input, output, opt=None):  pass

then “optional_data_names” should be as

>>> optional_data_names = ('opt',)

classmethod print_config(file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>) → None

abstract classmethod required_data_names(train: bool = True, inference: bool = False) → Tuple[str, ...]

Define the required names by Task

This function is used by

cls.check_task_requirements() If your model is defined as following,

>>> from espnet2.train.abs_espnet_model import AbsESPnetModel
>>> class Model(AbsESPnetModel):
...     def forward(self, input, output, opt=None):  pass

then “required_data_names” should be as

>>> required_data_names = ('input', 'output')

trainer

alias of Trainer