• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site
Master 2020/2021

Neural Networks and Deep Learning

Category 'Best Course for Career Development'
Category 'Best Course for Broadening Horizons and Diversity of Knowledge and Skills'
Category 'Best Course for New Knowledge and Skills'
Type: Elective course (Big Data Systems)
Area of studies: Business Informatics
When: 2 year, 1 module
Mode of studies: distance learning
Open to: students of all HSE University campuses
Instructors: Sergey Lisitsyn
Master’s programme: Big Data Systems
Language: English
ECTS credits: 6

Course Syllabus


Recently, deep learning has become a tool of choice for multiple problems in the fields of computer vision, robotics, biology, natural language processing, and many more. The technology is enabled by various milestones in computing and engineering, such as graphical processing units (GPU) and differentiable programming. Due to recent advances in open source software for deep learning, training a model has become a relatively simple exercise. However, the techniques used by researchers are fairly complicated. This course is focused on practical part of applying state-of-the-art techniques of deep learning on real-world problems. The course is based on popular and accessible MOOC called fast.ai and covers all the sections of the course with advice from machine learning practice.
Learning Objectives

Learning Objectives

  • Learn to train models that achieve state-of-the-art results in: computer vision, natural language processing, tabular data, and collaborative filtering
  • Learn approaches to turn models into web applications
  • Learn to implement deep learning training loop from the scratch
  • Acknowledge ethical implications of applying machine learning in practice
Expected Learning Outcomes

Expected Learning Outcomes

  • Knows the definitions of deep learning
  • Understands approaches to put machine learning system into production
  • Understands data ethics and able to detect ethical problems
  • Can construct a digit classifier using a deep learning model
  • Can solve multi-class and multi-label problems with deep learning
  • Knows and uses state-of-the-art approaches to train neural networks
  • Able to use embedding for tabular data and recommenders
  • Understands approaches to solve natural language processing problems
  • Understands the role of convolutions in image processing
  • Can construct recurrent neural network from scratch
  • Able to use residual blocks with neural networks
  • Knows advanced neural networks such as U-Net and Siamese
  • Able to use momentum and advanced optimizers for stochastic gradient descent
  • Can construct a neural network from scratch
Course Contents

Course Contents

  • Introduction
    A brief history of neural networks. Software for deep learning. Definitions and jargon.
  • Machine learning in production
    Limitations of deep learning. Creating an machine learning application.
  • Data ethics
    Feedback loops and bias. Fairness, accountability, and transparency.
  • Training a digit classifier
    Data representation. Gradient descent and loss functions.
  • Image classification
    Cross-entropy loss. Transfer learning.
  • Other computer vision problems
    Multi-label classification and regression for computer vision.
  • Reaching state-of-the-art
    Normalization and resizing. Label smoothing.
  • Collaborative filtering
    Embeddings. Bootstrapping a model.
  • Tabular data
    Categorical embeddings, decision trees.
  • Natural language processing
    Data representation. Recurrent neural networks.
  • Software libraries for data processing
    Mid-level API of the fastai library.
  • A language model from scratch
    Recurrent neural network building blocks. Long Short Term Memory (LSTM).
  • Convolutional neural networks
    Applying convolutions. Colored images and convolutions. Batch normalization.
  • Residual neural networks
    Using residual blocks to improve models.
  • Neural architectures
    U-Net and Siamese networks.
  • The training process of neural networks
    Baseline optimizer. Momentum and Adam.
  • A neural net from scratch
    Vector and matrix operations. Using PyTorch.
  • Class Activation Maps
    Using CAMs to evaluate models.
Assessment Elements

Assessment Elements

  • non-blocking Homework №1
    A student should provide a Jupyter notebook
  • non-blocking Homework №2
    A student should provide a Jupyter notebook
  • non-blocking Exam
    Exam format: the exam is taken in a written format, remotely (online) on MS Teams platform. A student is expected to complete the assignment and provide Jupyter (.ipynb) notebook to the instructor (the URL for Google Forms to submit is provided in MS Teams). To participate in the exam a student should have access to a compute that is either able to run Jupyter or work with Google Colab page. The file should be submitted one hour before the end of the exam. A student is allowed to use any kind of information to complete the assignment. A student should not consult or re-use solutions by other fellow students.
Interim Assessment

Interim Assessment

  • Interim assessment (1 module)
    0.2 * Exam + 0.4 * Homework №1 + 0.4 * Homework №2


Recommended Core Bibliography

  • Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). The Elements of Statistical Learning : Data Mining, Inference, and Prediction (Vol. Second edition, corrected 7th printing). New York: Springer. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsebk&AN=277008
  • Ian Goodfellow, Yoshua Bengio, & Aaron Courville. (2016). Deep Learning. The MIT Press.
  • Jeremy Howard, & Sylvain Gugger. (2020). Deep Learning for Coders with Fastai and PyTorch. O’Reilly Media.

Recommended Additional Bibliography

  • M Narasimha Murty, & V Susheela Devi. (2015). Introduction To Pattern Recognition And Machine Learning. World Scientific.