• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site
For visually-impairedUser profile (HSE staff only)SearchMenu

Deep Neural Network Ensembles: Analysis and Approaches to Diversification

Student: Aleksandr Lyzhov

Supervisor: Dmitry Vetrov

Faculty: Faculty of Computer Science

Educational Programme: Statistical Learning Theory (Master)

Year of Graduation: 2020

Complex machine learning models are increasingly used for safety-critical applications. Consequently, estimates of uncertainty in predictions play a big role in practical settings. In this work, we examine and create uncertainty estimation techniques for deep neural networks. We use image classification as a problem that neural networks solve, but the methods of uncertainty estimation can be easily adapted to other problems and types of data. We mostly focus on in-domain uncertainty estimation and attack the topic from several angles. First, we explore the standards of quantification of predictive uncertainty. We discuss drawbacks of metrics and the role of model calibration in uncertainty estimation. We also propose "deep ensemble equivalent" (DEE) score that measures uncertainty estimation relative to a baseline in a unified fashion to support interpretation of our results. Second, we conduct an empirical study of state-of-the-art methods for uncertainty estimation on a large scale. Specifically, we broadly evaluate large ensembles of deep neural networks. We interpret our findings in the light of properties of ensembling methods and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance. Third, we contribute to understanding and effectiveness of data augmentation. In recent years, methods of learning augmentation policies that are adapted to specific data and specific problems were proposed. Augmentation is commonly used for both training and inference, but these methods only focus on training. We show that existing learnable augmentation methods fail when straightforwardly applied to test-time augmentation. Nevertheless, we demonstrate that augmentation at test time can also be successfully learned. We introduce greedy policy search (GPS), a very simple but high-performing method for learning a policy of test-time augmentation. Policies learned with GPS achieve superior predictive performance, provide better in-domain uncertainty estimation, and improve the robustness to domain shift compared to alternatives. We demonstrate in an ablation that model calibration is an essential part of GPS.

Student Theses at HSE must be completed in accordance with the University Rules and regulations specified by each educational programme.

Summaries of all theses must be published and made freely available on the HSE website.

The full text of a thesis can be published in open access on the HSE website only if the authoring student (copyright holder) agrees, or, if the thesis was written by a team of students, if all the co-authors (copyright holders) agree. After a thesis is published on the HSE website, it obtains the status of an online publication.

Student theses are objects of copyright and their use is subject to limitations in accordance with the Russian Federation’s law on intellectual property.

In the event that a thesis is quoted or otherwise used, reference to the author’s name and the source of quotation is required.

Search all student theses