Outlier detection can be performed in unsupervised fashion if there are no labels, or supervised fashion if there is a training set where outliers have already been marked as such by an "expert".
Unsupervised outlier detection is often (always?) based on density. Algorithms will call observations outliers if they are too far away from most of the other observations. This is useful to detect cases which are likely to be either errors or exceptional cases; and eventually, for instance, to remove the errors from the dataset.
Supervised outlier detection is less common. Indeed, in real world applications, it is rather easy to determine what should be the normal situations (e.g. some equipment mode of operation), but quite complex to define abnormal situations. You could have examples of abnormal situations, but probably not all possible abnormal situations. Then, a supervised learning outlier detection (basically a binary classification problem with normal and abnormal classes) will learn how to detect known abnormal situations, but will likely fail on unknown ones.
This is why there is the semi-supervised outlier detection (called novelty detection in the scikit-learn example). The training dataset contains only normal points, the model is trained to define the boundaries of the normal domain. Then, at classification step, the algorithm can predict if an observation can be deemed normal or not. Semi-supervised outlier detection techniques are often based on the likelihood that observations were generated by the same unknown process which generated the learning dataset (see Bayesian networks or Gaussian mixture models for instance).
Notes:
- Unsupervised and semi-supervised techniques may be combined. For instance, you could use a first unsupervised approach to remove supposed outliers, then train a semi-supervised model on the remaining observations. The resulting model will likely be more robust than if outliers had been labelled as such, and a supervised model had been trained on the full dataset.
- For semi-supervised or unsupervised outlier detection, the decision boundary position depends on model hyperparameters (expected fraction of outliers, distance to inliers, acceptable likelihood, etc.)
