Эффективное обучение ансамблей глубоких нейронных сетей

Deep neural networks have achieved a number of state-of-the-art results on many real-life problems e.g. image classification and semantic segmentation. However, deep neural networks suffer from overfitting and overconfidence, because of a huge number of parameters. Ensembles of models are known to reduce these problems. Although the simple averaging of several independently trained deep neural networks may provides good results, it requires additional memory and computational resources. The approximate Bayesian inference allows to efficiently train ensembles of neural networks. However, in application to deep architectures Bayesian neural networks may suffer from instability during training. In our work, we propose Stochastic Batch Normalization — a new scalable, efficient and stable algorithm to ensemble deep neural networks. We build it on top of our new probabilistic interpretation of Batch Normalization technique. We demonstrate the performance of our method on modern architectures (including deep convolutional architectures: VGG-like and ResNets) for MNIST and CIFAR10 datasets and compare it with existing techniques.