espnet2.fileio.read_text.RandomTextReader

source

class espnet2.fileio.read_text.RandomTextReader(text_and_scp: str)

Bases: Mapping

Reader class for random access to text.

Simple text reader for non-pair text data (for unsupervised ASR) : Instead of loading the whole text into memory (often large for UASR), the reader consumes text which stores in byte-offset of each text file and randomly selected unpaired text from it for training using mmap.

Examples: : text : text1line text2line text3line <br/> scp : 11 00000000000000000010 00000000110000000020 00000000210000000030 <br/> scp explanation : (number of digits per int value) (text start at bytes 0 and end at bytes 10 (including “

“)) : (text start at bytes 11 and end at bytes 20 (including “

“)) : (text start at bytes 21 and end at bytes 30 (including “

“))

keys() → a set-like object providing a view on D's keys