Unsupervised Learning and Training Data

Question

As far as I know, we need to use training data to find out the relation between the features, also known as input values, and labels, that are output values, in supervised learning. After that, by using this relation, our learning system tries to predict labels of data samples in next data sets.

However, there is no need to find out such a relation in unsupervised learning because the data samples do not have labels; they only consist of features. In this case, do we need a training set in unsupervised learning ?

Answer 1

In unsupervised learning,the learning procedure is finding similarity between training samples, and putting similar items into a same cluster, training phase in unsupervised learning produce some sets with similar items. Then,in test phase the similarity is calculated for all items of each set, and check if the test item is similar to each cluster's items or not, if it was similar to at least one item, the test item belongs to that cluster.

SO... YES, we need training set and test set, the training set helps to find thresholds to find similarity

Answer 2

Premise: one usually says that in unsupervised learning no ground truth is available. Well, this is not always true. I would rather give a finer definition of unsupervised learning model: it is a model that, once you fix its hyper-parameters, does not use any ground truth to give its output.

If some ground truth is available and you build an unsupervised model, you just ignore this ground truth. Let me give you a practical example. Suppose you build an unsupervised anomaly detector, for instance based on k-means clustering (Ch. 6 of [1]). Suppose you have the ground truth, i.e., for some samples you know whether they are anomalous or normal. When K-means algorithm builds the cluster, it ignores this ground truth.

However, you need to establish a threshold tau, such that if a sample is distant from all cluster centroids more than tau, they you consider it an anomaly. How do you establish tau. One way to go is to plot how False Positives (i.e., False Alarms) and True Positives of anomaly detection change with different choices of tau and then choose a value for tau that seems ok. NOTE THAT, to compute False Positives and True Positives you need to use the ground truth.

So, although the method above is unsupervised, you do actually use the ground truth to select the hyper-parameter tau.

In this case, it would be a mistake (in particular a Data Leakage - Ch.8 of [2]) to choose the hyper-parameters on all your data and then evaluate the performance of your final anomaly detector on the same data again. Because in a real situation, you would not have the ground truth available to choose the hyper-parameters.

In such cases, the correct way to go is: - Divide your data in training/test set - Find the hyper-parameters using only the training set - Evaluate the performance on the test set

[1] Halder & Ozdemir, Hands on Machine Learning for Cyber-security, Packt, 2018

[2] Teo, J. (2019). Data Science Documentation. Retrieved from https://buildmedia.readthedocs.org/media/pdf/python-data-science/latest/python-data-science.pdf