espnet.lm.lm_utils.read_tokens

source

espnet.lm.lm_utils.read_tokens(filename, label_dict)

Read tokens as a sequence of sentences.

:param str filename : The name of the input file :param dict label_dict : dictionary that maps token label string to its ID number :return list of ID sequences :rtype list