Representations of Neural Networks for Effective Architecture Search

The majority of articles that solve the Neural Architecture Search (NAS) task are Reinforcement Learning-based and gradient-based methods. Other approaches are poorly researched by now. The main problem of the aforementioned main approaches is that they search for the best architecture within a discrete space of graphs and discrete optimization is a hard task. Meanwhile, if we can embed the neural network architectures into continuous representations, it would simplify the optimization. There are only a few articles that bring the task to continuous optimization, but I see this approach to be very prominent. In the thesis, I will suggest finding representations of the graphs using a variational autoencoder model (VAE) model. To this aim, I will check the quality of VAE representations for NAS and next optimize. In order not to spend a lot of time on re-training many architectures during experiments, I will use the largest model benchmark datasets named Nasbench101. The dataset includes a set of architectures trained on CIFAR10 and their training logs. Its search-space contains most of the popular architectures such as MobileNet, Inception, or Resnet. In chapter 1 you will find a review of the current state-of-the-art solutions of NAS problem, VAE approaches, and description of models datasets which I will consider. To check VAE’s ability to solve NAS problems, I divided the issue into 2 parts: in chapter 2 you will find research about whether we can predict scores from the embeddings directly and how many architectures we need to predict the final score with reasonable confidence, in chapter 3, I will optimize over the latent space to find a good architecture. In the paper, different VAE modifications will be considered and I will show that change of the optimization objective could highly improve VAE performance in the NAS task. A comparison of performance with competitors’ approaches will be at the end of the work.