Section outline
-
Course content: This course introduces basic as well as modern concepts of statistical learning in terms of artificial neural networks (deep learning), with applications in statistical data analysis. Topics treated include feedforward networks, regularization and optimization of networks with many layers, convolutional networks, recurrent networks and validation methods. The course may also include some of the following modern topics; autoencoders, representation learning, attention and transformers, information theoretic concepts of deep learning, explainable AI, etc.
Teaching
Teacher: Chun-Biu Li (cbli@math.su.se)
TA: Nik Tavakolian (nik.tavakolian@math.su.se)
Course literature: "Deep Learning" by Goodfellow et al., MIT Press, 2016
Course plan: 2022 Edition
Remarks: The lecture plan below may subject to change (add or skip some topics) during the semester depending on the progress of the class.
Examination
Examiner: Chun-Biu Li
Course Exam & Projects: The course consists of the following parts,
Part 1: (Theory 3.5 hp) There will be an open-book written exam on Mar 16, 2026. The exam covers some of the recommended exercises given in all classes. Registration for the exam in Ladok is required to take the exam.
Part 2: (Project 4 hp) 3 small projects - on materials in all lectures will be given as hand-in written assignments. 1 additional small project - two students work as a group pick up a dataset (see "Project 4" below for a list of databases), then analyze using any deep learning method with validations and present the result orally for the class (~20 mins) on Mar 9.
Examination Rules at the Department of Mathematics.
Grading Criteria: See below
Resources
-
Course Overview
Overview of deep learning and the structure of the course.
Mathematical Background
Reading: Sec 4.3, 5.4, 5.5, 5.9
Feedforward Networks
Reading: Sec 6.1-5
-
Continue on Feedforward Networks
Reading: Sec 6.1-5
-
Continue on Feedforward Networks
Reading: Sec 6.1-5
Regularization for DL
Reading: Sec 7.1, 7.3-5 (skip 7.2, 7.5.1), 7.7-8, 7.10-12 -
Continue on Regularization for DL
Reading: Sec 7.1, 7.3-5 (skip 7.5.1), 7.7-8, 7.10-12
-
Optimization
Reading: Sec 8.1-5 (Skip 8.3.3)
-
Continue on Optimization
Reading: Sec 8.1-5 (Skip 8.3.3)
-
Convolutional NN
Reading: Sec 9.1-9
-
Continue on Convolutional NN
Reading: Sec 9.1-9
-
Continue on Convolutional NN
Reading: Sec 9.1-9
Recurrent NN
Reading: 10.1-10 (skip 10.8)
Applications of RNN: https://iq.opengenus.org/applications-of-rnn/
-
Continue on Recurrent NN
Reading: 10.1-10 (skip 10.8)
-
Continue on Recurrent NN
Reading: 10.1-10 (skip 10.8)
Special Topic: Attention Mechanism and Transformers
Reading: Chapter 11 of "Dive into Deep Learning" (https://d2l.ai)
REMARKS: The book "Dive into Deep Learning" contains many state-of-the-art topics in DL that can be used as an excellent supporting reference. The book is also better written with more graphical illustrations compared to Goodfellow's book. Nevertheless, the "Dive" book doesn't provide enough mathematical and statistical backgrounds of DL and therefore is not recommended as the main textbook for the course. The best learning path is to first build up a firm background from Goodfellow's book and then move on to the "Dive" book to learn about the modern developments with better understanding.
-
Continue on Attention and Transformers
Reading: Chapter 11 of "Dive into Deep Learning" (https://d2l.ai)
-
Continue on Attention and Transformers
Reading: Chapter 11 of "Dive into Deep Learning" (https://d2l.ai)
Help Session -
Project 1 (on basics of neural networks and regularization)
Posting date: Jan 27
Due date: 8:00 pm Feb 9
- Completed by each individual student with a written report
-
Project 2 (on convolutional NN)
Posting date: Feb 9
Due date: 8:00 pm Feb 23
- Completed by each individual student with a written report
-
Project 3 (on recurrent NN or transformers)
Posting date: Feb 23
Due date: 8:00pm Mar 18
- Completed by each individual student with a written report
-
Project 4 (Practical applications)
Posting date: Jan 27
Report due date: Mar 9
Oral Presentation dates: Mar 9
Two students as a group pick up a dataset from a database or search engine, eg ,
1) Kaggle datasets (www.kaggle.com)
2) UCI ML repository (https://archive.ics.uci.edu/ml/index.php)
3) Google datasets search engine (https://datasetsearch.research.google.com)
4) Microsoft datasets (https://msropendata.com)
5) Awesome public datasets (https://github.com/awesomedata/awesome-public-datasets)
then analyze it using any deep learning method with suitable regularization and present the result orally for the class (~20 mins). A written report from the group will be submitted to the course page (see the report due date above).
Grading of your report and presentation of Project 4 will be based on the following criteria:
- Apply appropriate advanced deep learning method to the chosen dataset (25p)
- Display understanding of the method and knowledge about theoretical background (25p)
- Correctly interpret the analysis results (25p)
- Present the method and result in oral and report clearly and logically (25p)