espnet.nets.pytorch_backend.transformer.embedding.LearnableFourierPosEnc

source

class espnet.nets.pytorch_backend.transformer.embedding.LearnableFourierPosEnc(d_model, dropout_rate=0.0, max_len=5000, gamma=1.0, apply_scaling=False, hidden_dim=None)

Bases: Module

Learnable Fourier Features for Positional Encoding.

See https://arxiv.org/pdf/2106.02795.pdf

• Parameters:
  • d_model (int) – Embedding dimension.
  • dropout_rate (float) – Dropout rate.
  • max_len (int) – Maximum input length.
  • gamma (float) – init parameter for the positional kernel variance see https://arxiv.org/pdf/2106.02795.pdf.
  • apply_scaling (bool) – Whether to scale the input before adding the pos encoding.
  • hidden_dim (int) – if not None, we modulate the pos encodings with an MLP whose hidden layer has hidden_dim neurons.

Initialize class.

extend_pe(x)

Reset the positional encodings.

forward(x: Tensor)

Add positional encoding.

• Parameters:x (torch.Tensor) – Input tensor (batch, time, *).
• Returns: Encoded tensor (batch, time, *).
• Return type: torch.Tensor