espnet2.schedulers.warmup_lr.WarmupLR

source

class espnet2.schedulers.warmup_lr.WarmupLR(optimizer: Optimizer, warmup_steps: int | float = 25000, last_epoch: int = -1)

Bases: _LRScheduler, AbsBatchStepScheduler

The WarmupLR scheduler

This scheduler is almost same as NoamLR Scheduler except for following difference:

NoamLR: : lr = optimizer.lr * model_size ** -0.5 : * min(step ** -0.5, step * warmup_step ** -1.5)

WarmupLR: : lr = optimizer.lr * warmup_step ** 0.5 : * min(step ** -0.5, step * warmup_step ** -1.5)

Note that the maximum lr equals to optimizer.lr in this scheduler.

get_lr()