espnet.nets.chainer_backend.transformer.training.VaswaniRule

source

class espnet.nets.chainer_backend.transformer.training.VaswaniRule(attr, d, warmup_steps=4000, init=None, target=None, optimizer=None, scale=1.0)

Bases: Extension

Trainer extension to shift an optimizer attribute magically by Vaswani.

• Parameters:
  • attr (str) – Name of the attribute to shift.
  • rate (float) – Rate of the exponential shift. This value is multiplied to the attribute at each call.
  • init (float) – Initial value of the attribute. If it is None, the extension extracts the attribute at the first call and uses it as the initial value.
  • target (float) – Target value of the attribute. If the attribute reaches this value, the shift stops.
  • optimizer (Optimizer) – Target optimizer to adjust the attribute. If it is None, the main optimizer of the updater is used.

Initialize Vaswani rule extension.

initialize(trainer)

Initialize Optimizer values.

serialize(serializer)

Serialize extension.