ESPnet2

We are planning a super major update, called ESPnet2. The developing status is still under construction yet, so please be very careful to use with understanding following cautions:

  • There might be fatal bugs related to essential parts.

  • We haven’t achieved comparable results to espnet1 on each task yet.

Main changing from ESPnet1

  • Chainer free

    • Discarding Chainer completely.

    • The development of Chainer is stopped at v7: https://chainer.org/announcement/2019/12/05/released-v7.html

  • Kaldi free

    • It’s not mandatory to compile Kaldi.

    • If you find some recipes requiring Kaldi mandatory, please report it. It should be dealt with as a bug in ESPnet2.

    • We still support the features made by Kaldi optionally.

    • We still follow Kaldi style. i.e. depending on utils/ of Kaldi.

  • On the fly feature extraction & text preprocessing for training

    • You don’t need to create the feature file before training, but just input wave data directly.

    • We support both raw wave input and extracted features.

    • The preprocessing for text, tokenization to characters, or sentencepieces, can be also applied during training.

  • Discarding the JSON format describing the training corpus.

    • Why do we discard the JSON format? Because a dict object generated from a large JSON file requires much memory and it also takes much time to parse such a large JSON file.

  • Support distributed data-parallel training (Not enough tested)

    • Single node multi GPU training with DistributedDataParallel is also supported.

Recipes using ESPnet2

You can find the new recipes in egs2:

espnet/  # Python modules of epsnet1
espnet2/ # Python modules of epsnet2
egs/     # espnet1 recipes
egs2/    # espnet2 recipes

The usage of recipes is almost the same as that of ESPnet1.

  1. Change directory to the base directory

    # e.g.
    cd egs2/an4/asr1/
    

    an4 is a tiny corpus and can be freely obtained, so it might be suitable for this tutorial. You can perform any other recipes as the same way. e.g. wsj, librispeech, and etc.

    Keep in mind that all scripts should be ran at the level of egs2/*/{asr1,tts1,...}.

    # Doesn't work
    cd egs2/an4/
    ./asr1/run.sh
    ./asr1/scripts/<some-script>.sh
    
    # Doesn't work
    cd egs2/an4/asr1/local/
    ./data.sh
    
    # Work
    cd egs2/an4/asr1
    ./run.sh
    ./scripts/<some-script>.sh
    
  2. Change the configuration Describing the directory structure as follows:

    egs2/an4/asr1/
     - conf/      # Configuration files for training, inference, etc.
     - scripts/   # Bash utilities of espnet2
     - pyscripts/ # Python utilities of espnet2
     - steps/     # From Kaldi utilities
     - utils/     # From Kaldi utilities
     - db.sh      # The directory path of each corpora
     - path.sh    # Setup script for environment variables
     - cmd.sh     # Configuration for your backend of job scheduler
     - run.sh     # Entry point
     - asr.sh     # Invoked by run.sh
    
    • You need to modify db.sh for specifying your corpus before executing run.sh. For example, when you touch the recipe of egs2/wsj, you need to change the paths of WSJ0 and WSJ1 in db.sh.

    • Some corpora can be freely obtained from the WEB and they are written as “downloads/” at the initial state. You can also change them to your corpus path if it’s already downloaded.

    • path.sh is used to set up the environment for run.sh. Note that the Python interpreter used for ESPnet is not the current Python of your terminal, but it’s the Python which was installed at tools/. Thus you need to source path.sh to use this Python.

      . path.sh
      python
      
    • cmd.sh is used for specifying the backend of the job scheduler. If you don’t have such a system in your local machine environment, you don’t need to change anything about this file. See Using Job scheduling system

  3. Run run.sh

    ./run.sh
    

    run.sh is an example script, which we often call as “recipe”, to run all stages related to DNN experiments; data-preparation, training, and evaluation.

See training status

Show the log file

% tail -f exp/*_train_*/train.log
[host] 2020-04-05 16:34:54,278 (trainer:192) INFO: 2/40epoch started. Estimated time to finish: 7 minutes and 58.63 seconds
[host] 2020-04-05 16:34:56,315 (trainer:453) INFO: 2epoch:train:1-10batch: iter_time=0.006, forward_time=0.076, loss=50.873, los
s_att=35.801, loss_ctc=65.945, acc=0.471, backward_time=0.072, optim_step_time=0.006, lr_0=1.000, train_time=0.203
[host] 2020-04-05 16:34:58,046 (trainer:453) INFO: 2epoch:train:11-20batch: iter_time=4.280e-05, forward_time=0.068, loss=44.369
, loss_att=28.776, loss_ctc=59.962, acc=0.506, backward_time=0.055, optim_step_time=0.006, lr_0=1.000, train_time=0.173

Show the training status in a image file

# Accuracy plot
# (eog is Eye of GNOME Image Viewer)
eog exp/*_train_*/images/acc.img
# Attention plot
eog exp/*_train_*/att_ws/<sample-id>/<param-name>.img

Use tensorboard

tensorboard --logdir exp/*_train_*/tensorboard/

Instruction for run.sh

How to parse command-line arguments in shell scripts?

All shell scripts in espnet/espnet2 depend on utils/parse_options.sh to parase command line arguments.

e.g. If the script has ngpu option

#!/bin/bash
# run.sh
ngpu=1
. utils/parse_options.sh
echo ${ngpu}

Then you can change the value as follows:

$ ./run.sh --ngpu 2
echo 2

You can also show the help message:

./run.sh --help

Start from a specified stage and stop at a specified stage

The procedures in run.sh can be divided into some stages, e.g. data preparation, training, and evaluation. You can specify the starting stage and the stopping stage.

./run.sh --stage 2 --stop-stage 6

There are also some altenative otpions to skip specified stages:

run.sh --skip_data_prep true  # Skip data preparation stages.
run.sh --skip_train true      # Skip training stages.
run.sh --skip_eval true       # Skip decoding and evaluation stages.
run.sh --skip_upload false    # Enable packing and uploading stages.

Note that skip_upload is true by default. Please change it to false when uploading your model.

Change the configuration for training

Please keep in mind that run.sh is a wrapper script of several tools including DNN training command. You need to do one of the following two ways to change the training configuration.

# Give a configuration file
./run.sh --asr_train_config conf/train_asr.yaml
# Give arguments to "espnet2/bin/asr_train.py" directly
./run.sh --asr_args "--foo arg --bar arg2"

e.g. To change learning rate for the LM training

./run.sh --lm_args "--optim_conf lr=0.1"

This is the case of ASR training and you need to replace it accordingly for the other task. e.g. For TTS

./run.sh --tts_args "--optim_conf lr=0.1"

See Change the configuration for training for more detail about the usage of training tools.

Change the number of parallel jobs

./run.sh --nj 10             # Chnage the number of parallels for data preparation stages.
./run.sh --inference_nj 10   # Chnage the number of parallels for inference jobs.

We also support submitting jobs to multiple hosts to accelerate your experiment: See Using Job scheduling system

Multi GPUs training and distributed training

./run.sh --ngpu 4 # 4GPUs in a single node
./run.sh --ngpu 2 --num_nodes 2 # 2GPUs x 2nodes

Note that you need to setup your environment correctly to use distributed training. See the following two:

Use specified expereiment directory for evaluation

If you already have trained a model, you may wonder how to give it to run.sh when you’ll evaluate it later. By default the directory name is determined according to given options, asr_args, lm_args, or etc. You can overwrite it by --asr_exp and --lm_exp.

# For ASR recipe
./run.sh --skip_data_prep true --skip_train true --asr_exp <your_asr_exp_directory> --lm_exp <your_lm_exp_directory>

# For TTS recipe
./run.sh --skip_data_prep true --skip_train true --tts_exp <your_tts_exp_directory>

Evaluation without training using pretrained model

./run.sh --download_model <model_name> --skip_train true

You need to fill model_name by yourself. See the following link about our pretrain models: https://github.com/espnet/espnet_model_zoo