A Study of Parametrical Activation Functions in Neural Networks

The purpose of the research is to investigate practical differences between parametric and non-parametric activation functions in neural networks and determine an influence of parameters in parametric objective functions on the neural networks models accuracy. There is a plenty of methods to compare non-parametric and parametric activation functions. In the current work was decided to conduct a comparative analysis of Convolutional Neural Network model (CNN) with Parametric Rectified Linear Unit (PReLU) activation function against of CNN with non-parametric Rectified Linear Unit (ReLU) function. The research consists of building two identical models with various functions. According to scientific articles, PReLU enhances model fitting with almost zero extra computational complexity and little risk to overfit. Additionally, the usage of PReLU derives an initialization technique that particularly considers the rectifier nonlinearities. This way enables to train extremely deep rectified models directly from scratch and to explore more complicated models of neural networks. Both CNN models were trained on CIFAR-10 dataset, which was invented by Canadian Institute For Advanced Research. CIFAR-10 dataset is divided into 10 different classes, where each class consists of 6,000 color images of 32x32 shape (Recht, Roelofs, Schmidt, Shankar, 2018). The expected results are to gain a higher accuracy on CNN with PReLU rather than on CNN with ReLU and make the activation function parameter learnable for the whole network.