espnet2.asr.encoder.contextual_block_transformer_encoder.ContextualBlockTransformerEncoder

source

class espnet2.asr.encoder.contextual_block_transformer_encoder.ContextualBlockTransformerEncoder(input_size: int, output_size: int = 256, attention_heads: int = 4, linear_units: int = 2048, num_blocks: int = 6, dropout_rate: float = 0.1, positional_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.0, input_layer: str | None = 'conv2d', pos_enc_class=<class 'espnet.nets.pytorch_backend.transformer.embedding.StreamPositionalEncoding'>, normalize_before: bool = True, concat_after: bool = False, positionwise_layer_type: str = 'linear', positionwise_conv_kernel_size: int = 1, padding_idx: int = -1, block_size: int = 40, hop_size: int = 16, look_ahead: int = 16, init_average: bool = True, ctx_pos_enc: bool = True)

Bases: AbsEncoder

Contextual Block Transformer encoder module.

Details in Tsunoo et al. “Transformer ASR with contextual block processing” (https://arxiv.org/abs/1910.07204)

• Parameters:
  • input_size – input dim
  • output_size – dimension of attention
  • attention_heads – the number of heads of multi head attention
  • linear_units – the number of units of position-wise feed forward
  • num_blocks – the number of encoder blocks
  • dropout_rate – dropout rate
  • attention_dropout_rate – dropout rate in attention
  • positional_dropout_rate – dropout rate after adding positional encoding
  • input_layer – input layer type
  • pos_enc_class – PositionalEncoding or ScaledPositionalEncoding
  • normalize_before – whether to use layer_norm before the first block
  • concat_after – whether to concat attention layer’s input and output if True, additional linear will be applied. i.e. x -> x + linear(concat(x, att(x))) if False, no additional linear will be applied. i.e. x -> x + att(x)
  • positionwise_layer_type – linear of conv1d
  • positionwise_conv_kernel_size – kernel size of positionwise conv1d layer
  • padding_idx – padding_idx for input_layer=embed
  • block_size – block size for contextual block processing
  • hop_Size – hop size for block processing
  • look_ahead – look-ahead size for block_processing
  • init_average – whether to use average as initial context (otherwise max values)
  • ctx_pos_enc – whether to use positional encoding to the context vectors

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(xs_pad: Tensor, ilens: Tensor, prev_states: Tensor | None = None, is_final=True, infer_mode=False) → Tuple[Tensor, Tensor, Tensor | None]

Embed positions in tensor.

• Parameters:
  • xs_pad – input tensor (B, L, D)
  • ilens – input length (B)
  • prev_states – Not to be used now.
  • infer_mode – whether to be used for inference. This is used to distinguish between forward_train (train and validate) and forward_infer (decode).
• Returns: position embedded tensor and mask

forward_infer(xs_pad: Tensor, ilens: Tensor, prev_states: Tensor | None = None, is_final: bool = True) → Tuple[Tensor, Tensor, Tensor | None]

Embed positions in tensor.

• Parameters:
  • xs_pad – input tensor (B, L, D)
  • ilens – input length (B)
  • prev_states – Not to be used now.
• Returns: position embedded tensor and mask

forward_train(xs_pad: Tensor, ilens: Tensor, prev_states: Tensor | None = None) → Tuple[Tensor, Tensor, Tensor | None]

Embed positions in tensor.

• Parameters:
  • xs_pad – input tensor (B, L, D)
  • ilens – input length (B)
  • prev_states – Not to be used now.
• Returns: position embedded tensor and mask

output_size() → int