espnet2.fst.lm_rescore.remove_repeated_and_leq

source

espnet2.fst.lm_rescore.remove_repeated_and_leq(tokens: List[int], blank_id: int = 0)

Generate valid token sequence.

Result may be used as input of transformer decoder and neural language model. Fristly, remove repeated token from a “token alignment” seqs; Then remove blank symbols.

This function may be replaced by tokenizing word_seqs with tokenizer or composeing word_seqs_fsas with L_inv.fst or composing token_seqs with ctc_topo. Current method is slelected other than previous three methods because it won’t need an extra object, i.e. tokenizer, L.fst or ctc_topo.