Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated and tested on a data instances that contain outliers in them. An example of algorithms that can be used for novelty detection are one-class SVM (OCSVM) and Local Outlier Factor (LOF).
1 Answer
Answering your question: yes, depending on the hyperparameters you choose, you could overfit the considered normal data, if you fit your separating hyperplane between normal and novel points being too much "shaped" on your input data. There are, for instance in case of one-class support vector machines, some important hyperparams like nu or gamma:
- nu: with this one, you tell the oc-SVM the fraction of novel points (i.e. anomalies) you want to consider in your input data; this way, you don't overfit your model by not considering normal all the input data (it also depends on your use case, how sure you want to be about normality of the input data points...) You can test it with the scikit-learn package this way:
VS with nu=0.1, where you tell the model to consider a higher fraction of points as being abnormal:
so, the lower nu value is, the more you are "overfitting" your novelties detector (which could be better or worse depending on how well you know your input data).
- gamma: now, look at the effect of the value of gamma, which is crucial for overfitting your model:
with gamma=0.1 (and rbf kernel), you have the following decision surface:
VS with gamma = 10
where, with this last option, your are overfitting so much.
-
1$\begingroup$ the point here is that, as semi-supervised learning (sub-type os unsupervised), you cannot validate "as well" as with labels/values in supervised learning, so you must find a good balance as you beging having more "normal" data, so there is no a best rule in advance but a balance to look for $\endgroup$ Commented Sep 28, 2020 at 8:25
-
1$\begingroup$ I would check whether new considered normal data points are labeled as novelties or not, by using the so called anomaly score, which gives you a sense of how far these fall from the separating hyperplane; in a real use case I made in automotive industry, this was not easy to consider, as the experts were not sure about the normal conditions (so I wasn't either then) but i used an anomaly score to check if new points candidates of being "normal" were too far apart or not (by studying these scores distribution). BTW, please check the answer as valid (green tick) if you consider it valid :) $\endgroup$ Commented Sep 28, 2020 at 8:54
-
1$\begingroup$ I am a bit lost now. Do you mean to compare the anomaly score of each data instances in the validation set with the average anomaly score of the training set and see how far the points from the average? If they are far from each other then I am overfitting my model, is that right? $\endgroup$– s_amCommented Sep 28, 2020 at 9:07
-
1$\begingroup$ that seems to me a valid aproximation, but adding some criteria: for instance, check how many standard deviations is the score of a normal instance in the validation set from the mean score of the training set scores distribution. What I did in that real use case is to only consider as novel points the ones falling 2*sigma far from the mean score (because my model could send too many false positives by the nature of the system), but that can be different in your case, just experiment with it and validate if you can so you can adjust it $\endgroup$ Commented Sep 28, 2020 at 9:21
-
1$\begingroup$ that sounds a reasonable approach to me, but remember, you do not have that "working very well" certainty as in supervised learning, so you should rely on experience and real feedback to confirm it :) $\endgroup$ Commented Sep 28, 2020 at 11:47