espnet2.universa.base.universa_base.UniversaBase

source

class espnet2.universa.base.universa_base.UniversaBase(input_size: int, metric2id: Dict[str, int], use_ref_audio: bool = True, use_ref_text: bool = True, embedding_size: int = 512, use_normalize: bool = True, audio_encoder_type: str = 'transformer', audio_encoder_params: Dict[str, float | int | bool | str] = {'attention_dropout_rate': 0.1, 'attention_heads': 4, 'concat_after': False, 'dropout_rate': 0.1, 'input_layer': 'linear', 'layer_drop_rate': 0.0, 'linear_units': 2048, 'normalize_before': True, 'num_blocks': 3, 'positional_dropout_rate': 0.1, 'positionwise_conv_kernel_size': 1, 'positionwise_layer_type': 'linear', 'qk_norm': False, 'use_flash_attn': False}, vocab_size: int | None = None, ignore_id: int = -1, text_encoder_type: str = 'transformer', text_encoder_params: Dict[str, float | int | bool | str] = {'attention_dropout_rate': 0.1, 'attention_heads': 4, 'concat_after': False, 'dropout_rate': 0.1, 'input_layer': 'linear', 'layer_drop_rate': 0.0, 'linear_units': 2048, 'normalize_before': True, 'num_blocks': 3, 'positional_dropout_rate': 0.1, 'positionwise_conv_kernel_size': 1, 'positionwise_layer_type': 'linear', 'qk_norm': False, 'use_flash_attn': False}, cross_attention_type: str = 'multihead', cross_attention_params: Dict[str, float | int] = {'dropout_rate': 0.1, 'n_head': 4}, pooling_type: str = 'mean', pooling_params: Dict[str, float | int | bool | str] = {}, projector_type: str = 'linear', projector_params: Dict[str, float | int | bool | str] = {}, multi_branch: bool = False, use_mse: bool = False, use_l1: bool = True, metric_pad_value: float = -100, loss_weights: Dict[str | int, float] | None = None, **kwargs)

Bases: AbsUniversa

Initialize UniversaBase module.

• Parameters:
  • input_size (int) – Input dimension.
  • metric2id (Dict *[*str , int ]) – Metric to ID mapping.
  • vocab_size (Optional *[*int ]) – Number of vocabulary.
  • ignore_id (int) – Ignore ID.
  • use_ref_audio (bool) – Whether to use reference audio.
  • use_ref_text (bool) – Whether to use reference text.
  • embedding_size (int) – Embedding size.
  • use_normalize (bool) – Whether to normalize input features.
  • audio_encoder_type (str) – Audio encoder type.
  • audio_encoder_params (Dict *[*str , Sequence ]) – Audio encoder parameters.
  • text_encoder_type (str) – Text encoder type.
  • text_encoder_params (Dict *[*str , Sequence ]) – Text encoder parameters.
  • cross_attention_type (str) – Cross attention type.
  • cross_attention_params (Dict *[*str , Sequence ]) – Cross attention parameters.
  • pooling_type (str) – Pooling type.
  • pooling_params (Dict *[*str , Sequence ]) – Pooling parameters.
  • projector_type (str) – Projector type.
  • projector_params (Dict *[*str , Sequence ]) – Projector parameters.
  • multi_branch (bool) – Whether to use multi-branch pooling and projectors.
  • use_mse (bool) – Whether to use MSE loss.
  • use_l1 (bool) – Whether to use L1 loss.
  • metric_pad_value (float) – Metric padding value.
  • loss_weights (Optional *[*Dict *[*str , float ] ]) – Loss weights.

encode(audio: Tensor, audio_lengths: Tensor, ref_audio: Tensor | None = None, ref_audio_lengths: Tensor | None = None, ref_text: Tensor | None = None, ref_text_lengths: Tensor | None = None, **kwargs) → Tuple[Tensor, Tensor]

forward(audio: Tensor, audio_lengths: Tensor, metrics: Dict[str, Tensor], ref_audio: Tensor | None = None, ref_audio_lengths: Tensor | None = None, ref_text: Tensor | None = None, ref_text_lengths: Tensor | None = None, **kwargs) → Tuple[Tensor, Dict[str, Tensor], Tensor]

Calculate outputs and return the loss tensor.

• Parameters:
  • audio (torch.Tensor) – Input audio tensor (B, T).
  • audio_lengths (torch.Tensor) – Length of audio tensor (B,).
  • metrics (torch.Tensor) – Metrics tensor Dict[str, tensor (B,)].
  • ref_audio (torch.Tensor) – Reference audio tensor (B, T).
  • ref_audio_lengths (torch.Tensor) – Length of reference audio tensor (B,).
  • ref_text (torch.Tensor) – Reference text tensor (B, U).
  • ref_text_lengths (torch.Tensor) – Length of reference text tensor (B,).
• Returns: loss (torch.Tensor): Loss tensor. stats (Dict[str, torch.Tensor]): Statistics to be monitored. weight (torch.Tensor): Weight tensor.
• Return type: Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]

inference(audio: Tensor, audio_lengths: Tensor, ref_audio: Tensor | None = None, ref_audio_lengths: Tensor | None = None, ref_text: Tensor | None = None, ref_text_lengths: Tensor | None = None, **kwargs) → Dict[str, ndarray | Tensor]

Return predicted output as a dict.

• Parameters:
  • audio (torch.Tensor) – Input audio tensor (B, T).
  • audio_lengths (torch.Tensor) – Length of audio tensor (B,).
• Returns: Predicted output.
• Return type: Dict[str, torch.Tensor]