espnet2.asr.encoder.beats_encoder.quant_noise

source

espnet2.asr.encoder.beats_encoder.quant_noise(module, p, block_size)

Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described in “Training with Quantization Noise for Extreme Model Compression”

• Parameters:
  • module ( -) – nn.Module
  • p ( -) – amount of Quantization Noise
  • block_size ( -) – size of the blocks for subsequent quantization with iPQ

Remarks: : - Module weights must have the right sizes wrt the block size

• Only Linear, Embedding and Conv2d modules are supported for the moment
• For more detail on how to quantize by blocks with convolutional weights, see “And the Bit Goes Down: Revisiting the Quantization of Neural Networks”
• We implement the simplest form of noise here as stated in the paper which consists in randomly dropping blocks