espnet2.optimizers package

espnet2.optimizers.sgd

class espnet2.optimizers.sgd.SGD(params, lr: float = 0.1, momentum: float = 0.0, dampening: float = 0.0, weight_decay: float = 0.0, nesterov: bool = False)

Bases: torch.optim.sgd.SGD

Thin inheritance of torch.optim.SGD to bind the required arguments, ‘lr’

Note that the arguments of the optimizer invoked by AbsTask.main() must have default value except for ‘param’.

I can’t understand why only SGD.lr doesn’t have the default value.

espnet2.optimizers.__init__

espnet2.optimizers.optim_groups

espnet2.optimizers.optim_groups.add_optimizer_hooks(model, bias_weight_decay=False, normalization_weight_decay=False)

Set zero weight decay for some params

Set weight_decay=0.0 for parameters in model.no_weight_decay, for parameters with attribute _no_weight_decay==True, for bias parameters if bias_weight_decay==False, for normalization parameters if normalization_weight_decay==False

See: https://discuss.pytorch.org/t/weight-decay-only-for-weights-of-nn-linear-and-nn-conv/114348 # noqa

espnet2.optimizers.optim_groups.configure_optimizer(model, optim_class, optim_conf, weight_decay_conf)