espnet2.tok.quantizer.abs_quantizer.AbsSpeechTokenizerQuantizer

source

class espnet2.tok.quantizer.abs_quantizer.AbsSpeechTokenizerQuantizer(*args: Any, **kwargs: Any)

Bases: Module, ABC

Define the common contract for differentiable speech quantizers.

Implementations are expected to provide differentiable assignments for training, hard integer token IDs for inference, and the corresponding sequence lengths. This interface discretizes continuous features while preserving a gradient path for training. The similarly named speechlm.tokenizer.AbsTokenizer is intended for no-grad postprocessing of generated tokens, such as codec codes to waveform or BPE tokens to text. Use that interface for SpeechLM postprocessing and this one when discretization must preserve gradients.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

abstractmethod encode(features: Tensor, feature_lengths: Tensor) → SpeechTokenizerOutput

Quantize continuous features deterministically.

abstract property feature_dim : int

Return the expected dimension of input features.

abstractmethod forward(features: Tensor, feature_lengths: Tensor) → SpeechTokenizerOutput

Quantize continuous features with a differentiable assignment.

abstract property num_clusters : int

Return the number of discrete clusters.