espnet.scheduler.scheduler.CyclicCosineScheduler

source

class espnet.scheduler.scheduler.CyclicCosineScheduler(key: str, args: Namespace)

Bases: SchedulerInterface

Cyclic cosine annealing.

• Parameters:
  • cosine_warmup (int) – number of warmup iterations.
  • cosine_total (int) – number of total annealing iterations.

Notes

Proposed in https://openreview.net/pdf?id=BJYwwY9ll (and https://arxiv.org/pdf/1608.03983.pdf). Used in the GPT2 config of Megatron-LM https://github.com/NVIDIA/Megatron-LM

Initialize class.

alias = 'cosine'

scale(n_iter)

Scale of lr.