Lab Session 3 - Unsupervised Learning⚓
Note
Before you begin, make sure you have downloaded the latest update of the course slides from here, and keep them close while doing the lab.
Note
You may use any AI assistant to help you here (and at any moment in the project). Comprehension is what is graded though, so use them as helpers, never automatically and blindly!
Objectives of the lab⚓
At the end of this session, you will be able to:
- Cluster data with
KMeans, and choose \(K\) with the elbow and the silhouette heuristics. - Recognize when the assumptions of K-means break, and pick an algorithm that fits the shape of your data.
- Prepare your features so that distances actually mean something (scaling, removing useless features).
- Decompose data with
PCA: components, explained variance, reconstruction. - Visualize high dimensional data in 2D with
PCAandTSNE— and know what a t-SNE plot does not tell you. - Evaluate a result without ground truth, and with labels when they happen to exist.
- Run a first unsupervised exploration of your own project data.
The lab has three parts:
| Part 1 | ~1h | Toy datasets you generate yourself: the whole unsupervised toolbox on data whose answer you already know. |
| Part 2 | ~45 min | MNIST: the same tools on a real, high dimensional dataset (the one of the course slides). |
| Part 3 | ~45 min | Your project data: cluster it, visualize it, and confront the result with what you know. |
Note
Whenever possible, we have included links to the official documentation of the functions you need to use. Check them out, you'll save a lot of time!
Prerequisites⚓
You need the environment of lab 1 with scikit-learn installed in lab 2. Nothing new to install today (except, optionally, umap-learn at the very end).
From the previous labs, we assume you can: build a (N, d) numpy array from a dataset, make a scatter plot, and use the fit / predict API of sklearn.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
What changes with respect to lab 2⚓
Three things, and they matter for everything below:
- No
yinfit. Every model today is fitted withmodel.fit(X)alone. Labels, when they exist, are used only for display and evaluation, never for fitting. - No train/test split (see the course): there is no right answer to generalize to, so you fit on all the data you have.
- One exception, worth remembering: if an unsupervised step is a preprocessing for a supervised model (e.g. a PCA before a classifier), then it must be fitted on the training set only, otherwise information leaks from the test set.
fit_predictandfit_transformreplacepredictandscore: clustering methods assign a cluster index to each sample (fit_predict), decomposition and manifold methods produce new coordinates (fit_transform).
Table of contents of the lab⚓
Part 1 — The toolbox on data you control (~1h)
- Generate your toy datasets
- A first clustering with K-means
- Choosing \(K\): elbow and silhouette
- Distances have units
- Where K-means breaks
- Two dimensions are not enough: a controlled high dimensional example
Part 2 — A real dataset: MNIST (~45 min)
- Load and subsample MNIST
- Which pixels carry information?
- Clustering MNIST, and evaluating it
- Decomposition: PCA on MNIST
- Visualization: PCA vs t-SNE
Part 3 — Your own project data (~45 min)
Part 1 — The toolbox on data you control⚓
The point of Part 1 is that you decide what the structure of the data is, so you can tell whether an algorithm found it.
1.1 Generate your toy datasets⚓
Reuse make_blobs from lab 2, and add make_moons:
1 2 3 | |
1 2 3 | |
Important
Keep the "without colors" version in mind for the whole lab. Your algorithms only ever see the black-and-white version. y_blobs and y_moons exist here only so that you can check the answers.
1.2 A first clustering with K-means⚓
Use KMeans. The number of clusters is a parameter you must decide in advance:
1 2 | |
1 2 3 4 | |
Now compare your clusters to the true groups y_blobs. Do not use accuracy: cluster indices are arbitrary, cluster 0 has no reason to be class 0 (the course insists on this: a clustering is a partition, not a labelling). The metric that ignores this permutation is the Adjusted Rand Index:
1 | |
1 2 | |
Tip
K-means depends on its initialization (the course says: NP-hard problem, we only find a local minimum). n_init is the number of restarts sklearn does before keeping the best one. Try n_init=1 with a few different random_state and see how much the result moves. Keeping n_init=10 is a good default habit.
1.3 Choosing \(K\): elbow and silhouette⚓
In practice, nobody tells you the number of clusters. Compute both heuristics of the course over a range of \(K\):
- the inertia (
kmeans.inertia_), which always decreases with \(K\) → look for a bend, - the silhouette score (silhouette_score), bounded in \([-1, 1]\) → look for a maximum.
1 2 3 | |
Note
The silhouette needs at least 2 clusters (it compares each point to another cluster), which is why the range starts at 2. On a large dataset it costs \(O(N^2)\) distances, so use its sample_size argument.
1 | |
Important
You should already meet an uncomfortable case here. Both heuristics are definitions of what a good clustering is, and on the moons neither of them describes the structure you can see with your eyes. Being able to say "the metric says 2, but the metric is wrong here, and here is why" is exactly the skill this session is about.
1.4 Distances have units⚓
Every method of this session compares distances or variances. Neither is invariant to the scale of your features. Let's break a clustering that worked, by changing nothing but a unit:
1 2 | |
1 2 3 | |
You should observe a clear drop of the ARI, and a recovery after scaling. Ask yourself what this means for a real dataset with an age in years, a salary in euros and a distance in kilometers in the same matrix.
Tip
Chaining a scaler and a model is what Pipeline is made for: make_pipeline(StandardScaler(), KMeans(n_clusters=4)). Use it, it is one line and it prevents the classical mistake of scaling with the wrong statistics.
1.5 Where K-means breaks⚓
K-means only manipulates means and euclidean distances, so it implicitly looks for round, similarly sized clusters. The moons are neither.
1 | |
You should get an ARI around \(0.25\): the algorithm cuts the two moons in half instead of separating them, and no hyperparameter will fix this. The assumption is wrong, so the model must change. Try a density-based method, DBSCAN:
1 2 | |
Note that DBSCAN does not take a number of clusters: it grows clusters from dense neighborhoods, so the number of clusters is a result. Its two real hyperparameters are eps (the neighborhood radius) and min_samples. Points that belong to no dense region are labelled -1: they are noise, an option K-means does not have.
1 2 3 | |
Now the interesting part. Compute the silhouette score of both results (excluding the noise points for DBSCAN):
1 2 | |
Important
The silhouette prefers K-means, the solution you can see is wrong. This is not a bug: as stated in the course, the silhouette (like the inertia) rewards compact and convex clusters, so it encodes a definition of "good cluster" that the moons do not satisfy. A metric always answers one particular question. Whenever you report an internal metric in your project, be able to say which assumption it makes.
Call the teacher and discuss this comparison!
If you have time⚓
Build the other classical failure cases of K-means yourself, and find a model that handles them:
- Elongated clusters:
X @ [[0.6, -0.6], [-0.4, 0.8]]on blobs (anisotropic), then try GaussianMixture, which fits a covariance per cluster. - Clusters of very different sizes/densities:
make_blobs(..., cluster_std=[1.0, 2.5, 0.5]). - Other candidates worth 3 lines of code each: AgglomerativeClustering (try
linkage="single"on the moons) and SpectralClustering (tryaffinity="nearest_neighbors").
Tip
AgglomerativeClustering and SpectralClustering may be very useful for your project; don't hesitate to test them here!
The sklearn clustering comparison page shows all of this on one figure. Look at it after trying, not before.
1.6 Two dimensions are not enough: a controlled high dimensional example⚓
Everything above was 2D so you could plot it. Your project data will not be. Before using dimensionality reduction on a real dataset, let's check it on data whose structure we chose.
Case A — linear structure. Generate blobs in dimension 50, and project them in 2D with PCA:
1 2 3 4 5 | |
1 2 3 | |
Case B — curved structure. The "swiss roll" of the course slides:
1 2 | |
1 2 3 | |
Note
PCA only knows straight lines, so it can only photograph the roll from the best angle: layers that are far apart along the surface end up superimposed. t-SNE preserves neighborhoods instead, and recovers the 1D parameter almost exactly. This is the whole idea of manifold learning: the intrinsic dimension (here 1, up to the width of the roll) can be far below \(d\).
Part 2 — A real dataset: MNIST⚓
Same tools, on the dataset used throughout the course slides. You may reproduce some of the figures contained in the slides, so keep them open and close to you!
2.1 Load and subsample MNIST⚓
MNIST is 70000 handwritten digits of \(28 \times 28\) pixels, so \(d = 784\) once vectorized (see the course: the algorithms know nothing about the 2D layout of the pixels). We work on a random subset, both for speed and because t-SNE is \(O(N^2)\)-ish.
1 2 3 4 5 6 7 8 9 | |
Tip
The download takes a minute or two the first time (it is cached in ~/scikit_learn_data). Once you have saved the .npz, reload it with d = np.load("mnist_subset.npz"); X, y = d["X"], d["y"]. If the network is unreliable in the room, from sklearn.datasets import load_digits gives a much smaller \(8\times8\) version of the same problem, and everything below works with side = 8.
Here is the only plotting helper we provide; it displays any set of MNIST-sized vectors as a grid of images. You will reuse it for examples, masks, centroids, components and reconstructions.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | |
1 2 | |
2.2 Which pixels carry information?⚓
On MNIST, feature selection has an obvious meaning: some pixels are (almost) always black.
1 | |
1 2 | |
Now use VarianceThreshold to remove the least informative pixels. Rather than guessing a threshold, pick it as a percentile of the variances, so you directly control the proportion of removed features:
1 2 3 4 5 | |
1 2 3 4 5 6 | |
Important
Two lessons here. Selection destroys information, so it has to be justified. So, how would you justify selection here? Discuss this with a teacher!
2.3 Clustering MNIST, and evaluating it⚓
Compute KMEANS on MNIST:
1 | |
1 2 3 4 | |
Then evaluate, using the different metrics seen in the course:
1 2 3 4 | |
Important
Take the time to interpret the \(K=10\) vs \(K=20\) comparison, it is the heart of the course's evaluation section. With 20 clusters, homogeneity goes up (each cluster is purer) while completeness goes down (a digit is now split across several clusters), and the ARI does not necessarily improve. There is no contradiction: these numbers answer different questions. Compare visually the centroids to confirm this!
1 2 | |
Discuss your findings with a teacher!
2.4 Decomposition: PCA on MNIST⚓
1 | |
1 2 3 4 5 | |
Reconstruct, as on the slides. inverse_transform goes back to the pixel space, i.e. it computes \(UV\) of the course:
1 | |
1 2 3 4 5 | |
Note
Storing 32 coefficients instead of 784 pixels is a compression by a factor ~24, obtained without any label, and most (if not all) the digits stay readable. This is also why PCA is such a common preprocessing: going from \(d=784\) to \(d=32\) directly mitigates the curse of dimensionality of the previous session.
2.5 Visualization: PCA vs t-SNE⚓
Try to reproduce the last figure of the course, comparing PCA with t-SNE on MNIST. Fit the parameters to study their effects.
1 2 3 4 5 | |
1 2 | |
Important
The colors are the labels, used only for display: both algorithms ran without them. Doing exactly this is the cheapest sanity check of an unsupervised pipeline, and you will redo it on your own data in Part 3.
The caveats of the course are testable, so test them:
1 2 3 4 | |
If you have time⚓
- Does the decomposition help the clustering? Run
KMeans(n_clusters=10)onX_pca50instead ofX, and compare ARI and computation time. This is a downstream evaluation, the most honest kind according to the course. - UMAP:
pip install umap-learn, thenimport umap; umap.UMAP().fit_transform(X_pca50). Same usage as t-SNE, usually faster, and preserves a bit more of the global structure. - Clustering the t-SNE output: try it, then argue whether it is legitimate. (Hint: re-read the t-SNE caveats.)
Part 3 — Your own project data⚓
Apply the very same pipeline to your project dataset. This is exploratory work: the quality of your exploration and of your justifications will matter more than the beauty of any final plot.
-
Build your feature matrix. You need a numeric array
Xof shape(N, d), with no missing values (dropna, or fill them). If your data is not numeric yet (text, images, audio, categorical columns), extract the simplest reasonable features for now, as in lab 2; shallow features are fine today, better representations will come with foundation models in Session 5. -
Prepare your features, and say why. Are your columns in comparable units? If not, scale them (§1.4 showed what happens otherwise). Are some columns constant, or duplicated, or almost so?
-
Look at your data in 2D. Project with PCA, then with t-SNE, and color each plot with everything you have: your target if you have one, but also a category, a date, a source, the length of the sample. This is the highest-value 30 minutes of this lab for most projects: it is how you discover that your data is duplicated, unbalanced, dominated by one feature, or already separated. This step will still be useful for Deep Learning and Foundation models analysis!
-
How redundant are your features? Plot the cumulative explained variance. If 3 components out of 40 carry 95% of the variance, your 40 columns are measuring far fewer than 40 things.
-
Cluster it.
KMeanswith a \(K\) chosen with both heuristics (elbow and silhouette), plus a check of stability: re-run with severalrandom_stateand see whether the clusters move. Then try one non-K-means algorithm, chosen for a reason you can state (density-based if your 2D plot shows non-convex shapes, GMM if you see elongated groups, agglomerative if a hierarchy makes sense for your data). -
Evaluate, and interpret.
- Always available: an internal metric (silhouette), plus your own eyes on the 2D plot.
- If you have labels:
adjusted_rand_scoreandhomogeneity_completeness_v_measurefor evaluation only, never for fitting. - Then the real question: what did your clusters actually capture? Inspect a few samples of each cluster. If your clusters correspond to the recording device, the year, or the source file rather than to your target, that is a finding about your dataset (and a very common one).
Important
Conclude with one question: is this unsupervised step useful for my project, and for what? Acceptable answers include "the 2D projection showed that my two classes overlap completely with these features, so I need better ones", or "there is no cluster structure at this scale". A negative result that you correctly diagnose is worth more than a lucky one!
Note
Keep this notebook and these figures. A 2D projection of your data, colored by your target, may turn into a useful slide in your final presentation (or, conversely, may show the complexity of your task).
What you should take home from this session⚓
- The unsupervised API:
fit(X)alone, thenfit_predict/fit_transform, and labels only for evaluation. - Choosing \(K\) is a decision you justify: elbow and silhouette are heuristics that can disagree, and both can be wrong.
- No algorithm is best: K-means assumes round clusters, DBSCAN assumes dense regions, PCA assumes linear structure, t-SNE preserves only neighborhoods. Matching the assumption to the shape of your data is the actual work.
- Distances have units: scaling is not cosmetic, it may change the result.
- The reflex of projecting your data in 2D and coloring it with any metadata you have. Nonetheless, never trust blindly a vizualization; hyperparameters may introduce artifacts!
- A first unsupervised exploration of your own data.