Skip to main content

espnet2.svs.singing_tacotron.singing_tacotron.singing_tacotron

About 4 min

espnet2.svs.singing_tacotron.singing_tacotron.singing_tacotron

source

class espnet2.svs.singing_tacotron.singing_tacotron.singing_tacotron(idim: int, odim: int, midi_dim: int = 129, duration_dim: int = 500, embed_dim: int = 512, elayers: int = 1, eunits: int = 512, econv_layers: int = 3, econv_chans: int = 512, econv_filts: int = 5, atype: str = 'GDCA', adim: int = 512, aconv_chans: int = 32, aconv_filts: int = 15, cumulate_att_w: bool = True, dlayers: int = 2, dunits: int = 1024, prenet_layers: int = 2, prenet_units: int = 256, postnet_layers: int = 5, postnet_chans: int = 512, postnet_filts: int = 5, output_activation: str | None = None, use_batch_norm: bool = True, use_concate: bool = True, use_residual: bool = False, reduction_factor: int = 1, spks: int | None = None, langs: int | None = None, spk_embed_dim: int | None = None, spk_embed_integration_type: str = 'concat', use_gst: bool = False, gst_tokens: int = 10, gst_heads: int = 4, gst_conv_layers: int = 6, gst_conv_chans_list: Sequence[int] = (32, 32, 64, 64, 128, 128), gst_conv_kernel_size: int = 3, gst_conv_stride: int = 2, gst_gru_layers: int = 1, gst_gru_units: int = 128, dropout_rate: float = 0.5, zoneout_rate: float = 0.1, use_masking: bool = True, use_weighted_masking: bool = False, bce_pos_weight: float = 5.0, loss_type: str = 'L1', use_guided_attn_loss: bool = True, guided_attn_loss_sigma: float = 0.4, guided_attn_loss_lambda: float = 1.0)

Bases: AbsSVS

singing_Tacotron module for end-to-end singing-voice-synthesis.

This is a module of Spectrogram prediction network in Singing Tacotron described in

`Singing-Tacotron: Global Duration Control Attention and
Dynamic Filter for End-to-end Singing Voice Synthesis`_

, which learn accurate alignment information automatically.

Filter for End-to-end Singing Voice Synthesis`: : https://arxiv.org/pdf/2202.07907v1.pdf

Initialize Singing Tacotron module.

  • Parameters:
    • idim (int) – Dimension of the label inputs.
    • odim – (int) Dimension of the outputs.
    • embed_dim (int) – Dimension of the token embedding.
    • elayers (int) – Number of encoder blstm layers.
    • eunits (int) – Number of encoder blstm units.
    • econv_layers (int) – Number of encoder conv layers.
    • econv_filts (int) – Number of encoder conv filter size.
    • econv_chans (int) – Number of encoder conv filter channels.
    • dlayers (int) – Number of decoder lstm layers.
    • dunits (int) – Number of decoder lstm units.
    • prenet_layers (int) – Number of prenet layers.
    • prenet_units (int) – Number of prenet units.
    • postnet_layers (int) – Number of postnet layers.
    • postnet_filts (int) – Number of postnet filter size.
    • postnet_chans (int) – Number of postnet filter channels.
    • output_activation (str) – Name of activation function for outputs.
    • adim (int) – Number of dimension of mlp in attention.
    • aconv_chans (int) – Number of attention conv filter channels.
    • aconv_filts (int) – Number of attention conv filter size.
    • cumulate_att_w (bool) – Whether to cumulate previous attention weight.
    • use_batch_norm (bool) – Whether to use batch normalization.
    • use_concate (bool) – Whether to concat enc outputs w/ dec lstm outputs.
    • reduction_factor (int) – Reduction factor.
    • spks (Optional *[*int ]) – Number of speakers. If set to > 1, assume that the sids will be provided as the input and use sid embedding layer.
    • langs (Optional *[*int ]) – Number of languages. If set to > 1, assume that the lids will be provided as the input and use sid embedding layer.
    • spk_embed_dim (Optional *[*int ]) – Speaker embedding dimension. If set to > 0, assume that spembs will be provided as the input.
    • spk_embed_integration_type (str) – How to integrate speaker embedding.
    • use_gst (str) – Whether to use global style token.
    • gst_tokens (int) – Number of GST embeddings.
    • gst_heads (int) – Number of heads in GST multihead attention.
    • gst_conv_layers (int) – Number of conv layers in GST.
    • gst_conv_chans_list – (Sequence[int]): List of the number of channels of conv layers in GST.
    • gst_conv_kernel_size (int) – Kernel size of conv layers in GST.
    • gst_conv_stride (int) – Stride size of conv layers in GST.
    • gst_gru_layers (int) – Number of GRU layers in GST.
    • gst_gru_units (int) – Number of GRU units in GST.
    • dropout_rate (float) – Dropout rate.
    • zoneout_rate (float) – Zoneout rate.
    • use_masking (bool) – Whether to mask padded part in loss calculation.
    • use_weighted_masking (bool) – Whether to apply weighted masking in loss calculation.
    • bce_pos_weight (float) – Weight of positive sample of stop token (only for use_masking=True).
    • loss_type (str) – Loss function type (“L1”, “L2”, or “L1+L2”).
    • use_guided_attn_loss (bool) – Whether to use guided attention loss.
    • guided_attn_loss_sigma (float) – Sigma in guided attention loss.
    • guided_attn_loss_lambda (float) – Lambda in guided attention loss.

forward(text: Tensor, text_lengths: Tensor, feats: Tensor, feats_lengths: Tensor, label: Dict[str, Tensor] | None = None, label_lengths: Dict[str, Tensor] | None = None, melody: Dict[str, Tensor] | None = None, melody_lengths: Dict[str, Tensor] | None = None, duration: Dict[str, Tensor] | None = None, duration_lengths: Dict[str, Tensor] | None = None, pitch: Tensor | None = None, pitch_lengths: Tensor | None = None, slur: LongTensor | None = None, slur_lengths: Tensor | None = None, ying: Tensor | None = None, spembs: Tensor | None = None, sids: Tensor | None = None, lids: Tensor | None = None, joint_training: bool = False, flag_IsValid=False) → Tuple[Tensor, Dict[str, Tensor], Tensor]

Calculate forward propagation.

  • Parameters:

    • text (LongTensor) – Batch of padded character ids (B, T_text).

    • text_lengths (LongTensor) – Batch of lengths of each input batch (B,).

    • feats (Tensor) – Batch of padded target features (B, T_feats, odim).

    • feats_lengths (LongTensor) –

      Batch of the lengths of each target (B,). : label (Optional[Dict]): key is “lab” or “score”;

      value (LongTensor): Batch of padded label ids (B, Tmax).

    • label_lengths (Optional *[*Dict ]) – key is “lab” or “score”; value (LongTensor): Batch of the lengths of padded label ids (B, ).

    • melody (Optional *[*Dict ]) – key is “lab” or “score”; value (LongTensor): Batch of padded melody (B, Tmax).

    • melody_lengths (Optional *[*Dict ]) – key is “lab” or “score”; value (LongTensor): Batch of the lengths of padded melody (B, ).

    • pitch (FloatTensor) – Batch of padded f0 (B, Tmax).

    • pitch_lengths (LongTensor) – Batch of the lengths of padded f0 (B, ).

    • duration (Optional *[*Dict ]) – key is “lab”, “score_phn” or “score_syb”; value (LongTensor): Batch of padded duration (B, Tmax).

    • duration_length (Optional *[*Dict ]) – key is “lab”, “score_phn” or “score_syb”; value (LongTensor): Batch of the lengths of padded duration (B, ).

    • slur (LongTensor) – Batch of padded slur (B, Tmax).

    • slur_lengths (LongTensor) – Batch of the lengths of padded slur (B, ).

    • spembs (Optional *[*Tensor ]) – Batch of speaker embeddings (B, spk_embed_dim).

    • sids (Optional *[*Tensor ]) – Batch of speaker IDs (B, 1).

    • lids (Optional *[*Tensor ]) – Batch of language IDs (B, 1).

    • joint_training (bool) – Whether to perform joint training with vocoder.

  • Returns: Loss scalar value. Dict: Statistics to be monitored. Tensor: Weight value if not joint training else model outputs.
  • Return type: Tensor

inference(text: Tensor, feats: Tensor | None = None, label: Dict[str, Tensor] | None = None, melody: Dict[str, Tensor] | None = None, duration: Dict[str, Tensor] | None = None, slur: Dict[str, Tensor] | None = None, pitch: Tensor | None = None, spembs: Tensor | None = None, sids: Tensor | None = None, lids: Tensor | None = None, threshold: float = 0.5, minlenratio: float = 0.0, maxlenratio: float = 30.0, use_att_constraint: bool = False, use_dynamic_filter: bool = False, backward_window: int = 1, forward_window: int = 3, use_teacher_forcing: bool = False) → Dict[str, Tensor]

Generate the sequence of features given the sequences of characters.

  • Parameters:
    • text (LongTensor) – Input sequence of characters (T_text,).
    • feats (Optional *[*Tensor ]) – Feature sequence to extract style (N, idim).
    • label (Optional *[*Dict ]) – key is “lab” or “score”; value (LongTensor): Batch of padded label ids (Tmax).
    • melody (Optional *[*Dict ]) – key is “lab” or “score”; value (LongTensor): Batch of padded melody (Tmax).
    • pitch (FloatTensor) – Batch of padded f0 (Tmax).
    • duration (Optional *[*Dict ]) – key is “lab”, “score_phn” or “score_syb”; value (LongTensor): Batch of padded duration (Tmax).
    • slur (LongTensor) – Batch of padded slur (B, Tmax).
    • spembs (Optional *[*Tensor ]) – Speaker embedding (spk_embed_dim,).
    • sids (Optional *[*Tensor ]) – Speaker ID (1,).
    • lids (Optional *[*Tensor ]) – Language ID (1,).
    • threshold (float) – Threshold in inference.
    • minlenratio (float) – Minimum length ratio in inference.
    • maxlenratio (float) – Maximum length ratio in inference.
    • use_att_constraint (bool) – Whether to apply attention constraint.
    • use_dynamic_filter (bool) – Whether to apply dynamic filter.
    • backward_window (int) – Backward window in attention constraint or dynamic filter.
    • forward_window (int) – Forward window in attention constraint or dynamic filter.
    • use_teacher_forcing (bool) – Whether to use teacher forcing.
  • Returns: Output dict including the following items: : * feat_gen (Tensor): Output sequence of features (T_feats, odim).
    • prob (Tensor): Output sequence of stop probabilities (T_feats,).
    • att_w (Tensor): Attention weights (T_feats, T).
  • Return type: Dict[str, Tensor]
Copyright © 2024 ESPnet Community. All rights reserved.