• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site

Bayesian Pruning of Neural Networks

Student: Kharitonov Valerii

Supervisor: Dmitry Vetrov

Faculty: Faculty of Computer Science

Educational Programme: Data Science (Master)

Year of Graduation: 2018

We study the empirical Bayes approach to hyperparameter optimization and model selection in Bayesian neural networks. We explore its automatic relevance determination and pruning effect when Gaussian prior and approximate posterior for neural network weights are used. The connection is drawn between Sparse Variational Dropout and Gaussian Automatic Relevance Determination objectives. Based on the empirical Bayes procedure, we propose a novel type of prior that encourages structured sparsity and allows for compression and speeding up of modern convolutional deep neural network architectures. We provide experimental results showing that the proposed structured pruning procedure removes most of the filters or neurons in convolutional and dense layers in LeNet5 and VGG-like convolutional neural networks with almost no drop in accuracy of classification on MNIST and CIFAR-10 datasets. The new approach outperforms previously reported results in terms of sparsity levels in convolutional layers.

Student Theses at HSE must be completed in accordance with the University Rules and regulations specified by each educational programme.

Summaries of all theses must be published and made freely available on the HSE website.

The full text of a thesis can be published in open access on the HSE website only if the authoring student (copyright holder) agrees, or, if the thesis was written by a team of students, if all the co-authors (copyright holders) agree. After a thesis is published on the HSE website, it obtains the status of an online publication.

Student theses are objects of copyright and their use is subject to limitations in accordance with the Russian Federation’s law on intellectual property.

In the event that a thesis is quoted or otherwise used, reference to the author’s name and the source of quotation is required.

Search all student theses