Section outline
-
Course contents: This course introduces basic and modern concepts of statistical learning without training labels - unsupervised statistical learning, with applications in statistical data analysis. Central concepts covered include similarity measures, linear and nonlinear methods of dimensional reduction, combinatorial, distributional and density-based methods of cluster analysis, hierarchical methods and different validation methods.
Teaching
Teacher: Chun-Biu Li (cbli@math.su.se);
TA: Nik Tavakolian (nik.tavakolian@math.su.se)
Course literature: A few textbooks are available under "Resources", modern topics will be covered in terms of journal papers.
Course plan: 2026 Edition
Remarks: The lecture plan below may subject to change (add or skip some topics) during the semester depending on the progress of the class.
Examination
Examiner: Chun-Biu Li
Course Exam & Projects: The course consists of the following parts,
Part 1: (Theory 3 hp) There will be an open-book written exam on Oct 21, 2024. The exam covers some of the recommended exercises given in all classes. Registration for the exam in Ladok is required to take the exam.
Part 2: (Project 4.5 hp) 3 small projects - on materials in all lectures will be given as hand-in written assignments. 1 additional small project - two students work as a group pick up a dataset (see "Project 4" below for a list of databases), then analyze using any unsupervised learning method with validations and present the result orally for the class (~20 mins) on DATE TBD.
Examination Rules at the Department of Mathematics.
Grading Criteria: See below
Resources
-
Course Overview
Overview of unsupervised learning and the structure of the course.
Note: Unsupervised learning is a wide and rapidly developing field that it is impossible to cover all aspects in this course. E.g., we are not able to cover neural network based methods (will cover in the course "Statistical Deep Learning" in HT27), information theory based methods (will cover in the course "Statistical Information Theory"), fuzzy clustering, community detection in empirical networks, topological based methods, practical issues to handle large dataset, extensive discussion of validation methods for dimensionality reduction, generative models, etc. If you are interested in those advance topics, you can contact me (Chun) in person.
Basic Concepts in Clustering
a) Similarity and Dissimilarity measures
Reading: Sec 14.3.1-3 in ESL
b) Loss function for clustering
Reading: Sec 14.3.4-5 in ESL
c) k-mean
Reading: Sec 9.1 in PRML
-
Continue on Basic concepts in Clustering
d) Gaussian mixtures as soft k-mean clustering, EM for Gaussian mixtures
Reading: Sec 9.2 and 9.3.2 in PRML
e) k-medoids
Reading: Sec 14.3.10 in ESL
-
Continue on Basic Concepts in Clustering
f) Hierarchical clustering
Reading: Sec 14.3.12 in ESL
Basic Concepts in Dimensionality Reduction
a) Metric, distance and scalar product
Reading: Sec 4.2.1 in NDR
b) Principal component analysis (PCA)
Reading: Sec 2.4.1-2 (up to Eq. 2.19) in NDR
-
Continue on Basic Concepts in Dimensionality Reduction
c) Multidimensional scaling (MDS) and variants
Reading: Sec 4.2.2 in NDR
If time permits, I will also cover Sammon mapping, reading: Sec 4.2.3 in NDR
-
Continue on Basic Concepts in Dimensionality Reduction
d) Kernel PCA
Reading: Sec 4.4.1 in NDR
-
Validation Methods
Materials: Lecture notes
-
Modern Methods: Graph-based Methods
a) From data to graph
Reading: Sec 1-2 of Von Luxburg 2006
b) Graph Laplacians
Reading: Sec 3 of Von Luxburg 2006
c) Spectral clustering
Reading: Sec 4 of Von Luxburg 2006
-
Continue on Graph-based Methods
d) Graph cuts
Reading: Sec 5 of Von Luxburg 2006
e) Random walk on graph, commute-time distances and practical issues
Reading: Sec 6 and 8 of Von Luxburg 2006
-
Continue on Graph-based Methods
f) Laplacian eigenmaps
Reading: Sec 1, 2, 3.1, 4, 6 of Belkin & Niyogi 2003
If time permits, I will also discuss Diffusion Maps briefly, reading: de la Porte et al. 2008
-
Modern Methods: More Nonlinear Dimensionality Reduction
a) ISOMAP
Reading: Tenenbaum et al. 2000
b) Local Linear Embedding (LLE)
Reading: Roweis & Saul 2000
-
Continue on Nonlinear Dimensionality Reduction
b) Continue on LLE and its relation to graph Laplacian
c) Self-organizing maps
Reading: Sec 14.4 in ESL
Reading: Sec 5 of Belkin & Niyogi 2003
-
Density-based Methods for Clustering
a) DBSCAN
Reading: Ester et al. 1996
b) Clustering by fast search and find of density peaks
Reading: Rodriguez & Laio 2014
-
Special Topics in Unsupervised Learning
I will first talk about tSNE, and Nik (your TA) will continue on UMAP and beyond...
tSNE:
https://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf
UMAP:
https://joss.theoj.org/papers/10.21105/joss.00861 -
Project 1 (on basics of clustering, dim reduction and validations)
Posting date: TBD
Due date: TBD
- Completed by each individual student with a written report
-
Project 2 (on graph-based methods)
Posting date: TBD
Due date: TBD
- Completed by each individual student with a written report
-
Project 3 (on nonlinear dim reductions and/or density-based clustering)
Posting date: TBD
Due date: TBD
- Completed by each individual student with a written report
-
Project 4 (Practical applications)
Posting date: TBD
Oral Presentation date: TBD
Two students as a group pick up a dataset from one of the following databases or search engines,
1) Kaggle datasets (www.kaggle.com)
2) UCI ML repository (https://archive.ics.uci.edu/ml/index.php)
3) Google datasets search engine (https://datasetsearch.research.google.com)
4) Microsoft datasets (https://msropendata.com)
5) Awesome public datasets (https://github.com/awesomedata/awesome-public-datasets)
6) Your own research data
then analyze it using any unsupervised learning method with validations and present the result orally (~20 mins). A written report from the group will be handed in before TBD
Grading of your report and presentation of project 4 will be based on the following criteria:
- Apply appropriate advanced unsupervised learning method to the chosen dataset (25p)
- Display understanding of the method and knowledge about theoretical background (25p)
- Correctly interpret the analysis results (25p)
- Present the method and result in oral and report clearly and logically (25p)