espnet.utils.training.batchfy.make_batchset

source

espnet.utils.training.batchfy.make_batchset(data, batch_size=0, max_length_in=inf, max_length_out=inf, num_batches=0, min_batch_size=1, shortest_first=False, batch_sort_key='input', swap_io=False, mt=False, count='auto', batch_bins=0, batch_frames_in=0, batch_frames_out=0, batch_frames_inout=0, iaxis=0, oaxis=0)

Make batch set from json dictionary.

if utts have “category” value,

>>> data = {'utt1': {'category': 'A', 'input': ...},
...         'utt2': {'category': 'B', 'input': ...},
...         'utt3': {'category': 'B', 'input': ...},
...         'utt4': {'category': 'A', 'input': ...}}
>>> make_batchset(data, batchsize=2, ...)
[[('utt1', ...), ('utt4', ...)], [('utt2', ...), ('utt3': ...)]]

Note that if any utts doesn’t have “category”, perform as same as batchfy_by_

• Parameters:
  • data (Dict *[*str , Dict *[*str , Any ] ]) – dictionary loaded from data.json
  • batch_size (int) – maximum number of sequences in a minibatch.
  • batch_bins (int) – maximum number of bins (frames x dim) in a minibatch.
  • batch_frames_in (int) – maximum number of input frames in a minibatch.
  • batch_frames_out (int) – maximum number of output frames in a minibatch.
  • batch_frames_out – maximum number of input+output frames in a minibatch.
  • count (str) – strategy to count maximum size of batch. For choices, see espnet.asr.batchfy.BATCH_COUNT_CHOICES
  • max_length_in (int) – maximum length of input to decide adaptive batch size
  • max_length_out (int) – maximum length of output to decide adaptive batch size
  • num_batches (int) – # number of batches to use (for debug)
  • min_batch_size (int) – minimum batch size (for multi-gpu)
  • shortest_first (bool) – Sort from batch with shortest samples to longest if true, otherwise reverse
  • batch_sort_key (str) – how to sort data before creating minibatches [“input”, “output”, “shuffle”]
  • swap_io (bool) – if True, use “input” as output and “output” as input in data dict
  • mt (bool) – if True, use 0-axis of “output” as output and 1-axis of “output” as input in data dict
  • iaxis (int) – dimension to access input (for ASR, TTS iaxis=0, for MT iaxis=”1”.)
  • oaxis (int) – dimension to access output (for ASR, TTS, MT oaxis=0, reserved for future research, -1 means all axis.)
• Returns: List[List[Tuple[str, dict]]] list of batches