espnet.lm.lm_utils.ParallelSentenceIterator

source

class espnet.lm.lm_utils.ParallelSentenceIterator(dataset, batch_size, max_length=0, sos=0, eos=0, repeat=True, shuffle=True)

Bases: Iterator

Dataset iterator to create a batch of sentences.

This iterator returns a pair of sentences, where one token is shifted between the sentences like ‘<sos> w1 w2 w3’ and ‘w1 w2 w3 <eos>’ Sentence batches are made in order of longer sentences, and then randomly shuffled.

Initialize class.

property epoch_detail

Return epoch detail.

property previous_epoch_detail

Return previous epoch detail.

serialize(serializer)

Append values to serializer.

start_shuffle()

Shuffle batch indices.