Batch Normalisation

Batch normalisation aims to reduce internal covariate shift, and in doing so aims to accelerate the training of deep neural nets. It accomplishes this via a normalisation step that fixes the means and variances of layer inputs. Batch normalisation also has a beneficial effect on the gradient flow through the network, by reducing the dependence of gradients on the scale of the parameters or of their initial values. This allows for use of much higher learning rates without the risk of divergence. Furthermore, batch normalisation regularises the model and reduces the need for dropout.

We apply a batch normalisation layer as follows for a mini-batch $B$:

$$\begin{array}{r}{\mu }_B=\frac{1}{m}\sum _{i=1}^m{x}_i\\ {\sigma }_B^2=\frac{1}{m}\sum _{i=1}^m(x_i-{\mu }_B{)}^2\\ {\hat{x}}_i=\frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}\\ y_i=\gamma {\hat{x}}_i+\beta ={\text{BN}}_{\gamma ,\beta }(x_i)\end{array}$$

Where $\gamma$ and $\beta$ are learnable parameters.

References

• Papers With Code