Master
2020/2021
Neural Networks and Deep Learning
Category 'Best Course for Career Development'
Category 'Best Course for Broadening Horizons and Diversity of Knowledge and Skills'
Category 'Best Course for New Knowledge and Skills'
Type:
Elective course
Area of studies:
Business Informatics
Delivered by:
Department of Business Informatics
Where:
Graduate School of Business
When:
2 year, 1 module
Mode of studies:
distance learning
Instructors:
Sergey Lisitsyn
Master’s programme:
Big Data Systems
Language:
English
ECTS credits:
6
Contact hours:
28
Course Syllabus
Abstract
Recently, deep learning has become a tool of choice for multiple problems in the fields of computer vision, robotics, biology, natural language processing, and many more. The technology is enabled by various milestones in computing and engineering, such as graphical processing units (GPU) and differentiable programming. Due to recent advances in open source software for deep learning, training a model has become a relatively simple exercise. However, the techniques used by researchers are fairly complicated. This course is focused on practical part of applying state-of-the-art techniques of deep learning on real-world problems. The course is based on popular and accessible MOOC called fast.ai and covers all the sections of the course with advice from machine learning practice.
Learning Objectives
- Learn to train models that achieve state-of-the-art results in: computer vision, natural language processing, tabular data, and collaborative filtering
- Learn approaches to turn models into web applications
- Learn to implement deep learning training loop from the scratch
- Acknowledge ethical implications of applying machine learning in practice
Expected Learning Outcomes
- Knows the definitions of deep learning
- Understands approaches to put machine learning system into production
- Understands data ethics and able to detect ethical problems
- Can construct a digit classifier using a deep learning model
- Can solve multi-class and multi-label problems with deep learning
- Knows and uses state-of-the-art approaches to train neural networks
- Able to use embedding for tabular data and recommenders
- Understands approaches to solve natural language processing problems
- Understands the role of convolutions in image processing
- Can construct recurrent neural network from scratch
- Able to use residual blocks with neural networks
- Knows advanced neural networks such as U-Net and Siamese
- Able to use momentum and advanced optimizers for stochastic gradient descent
- Can construct a neural network from scratch
Course Contents
- IntroductionA brief history of neural networks. Software for deep learning. Definitions and jargon.
- Machine learning in productionLimitations of deep learning. Creating an machine learning application.
- Data ethicsFeedback loops and bias. Fairness, accountability, and transparency.
- Training a digit classifierData representation. Gradient descent and loss functions.
- Image classificationCross-entropy loss. Transfer learning.
- Other computer vision problemsMulti-label classification and regression for computer vision.
- Reaching state-of-the-artNormalization and resizing. Label smoothing.
- Collaborative filteringEmbeddings. Bootstrapping a model.
- Tabular dataCategorical embeddings, decision trees.
- Natural language processingData representation. Recurrent neural networks.
- Software libraries for data processingMid-level API of the fastai library.
- A language model from scratchRecurrent neural network building blocks. Long Short Term Memory (LSTM).
- Convolutional neural networksApplying convolutions. Colored images and convolutions. Batch normalization.
- Residual neural networksUsing residual blocks to improve models.
- Neural architecturesU-Net and Siamese networks.
- The training process of neural networksBaseline optimizer. Momentum and Adam.
- A neural net from scratchVector and matrix operations. Using PyTorch.
- Class Activation MapsUsing CAMs to evaluate models.
Assessment Elements
- Homework №1A student should provide a Jupyter notebook
- Homework №2A student should provide a Jupyter notebook
- ExamExam format: the exam is taken in a written format, remotely (online) on MS Teams platform. A student is expected to complete the assignment and provide Jupyter (.ipynb) notebook to the instructor (the URL for Google Forms to submit is provided in MS Teams). To participate in the exam a student should have access to a compute that is either able to run Jupyter or work with Google Colab page. The file should be submitted one hour before the end of the exam. A student is allowed to use any kind of information to complete the assignment. A student should not consult or re-use solutions by other fellow students.
Bibliography
Recommended Core Bibliography
- Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). The Elements of Statistical Learning : Data Mining, Inference, and Prediction (Vol. Second edition, corrected 7th printing). New York: Springer. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsebk&AN=277008
- Ian Goodfellow, Yoshua Bengio, & Aaron Courville. (2016). Deep Learning. The MIT Press.
- Jeremy Howard, & Sylvain Gugger. (2020). Deep Learning for Coders with Fastai and PyTorch. O’Reilly Media.
Recommended Additional Bibliography
- M Narasimha Murty, & V Susheela Devi. (2015). Introduction To Pattern Recognition And Machine Learning. World Scientific.