0
$\begingroup$

I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a naive and problematic approach. For example, say you have many individuals with a 'gender' label and an 'income' label. Also assume that there are many more men in the dataset than women. Unfortunately, due to income disparity we may see that women receive a lower wage than men, so if we were to simply plot a boxplot on the income feature and remove outliers we wouldn't be taking into account that some of those datapoints come from a different group (and furthermore, the assumption of more men than women means that we would likely remove a lot of the women from the dataset).

It seems like a better approach would be to remove outliers on a group-by-group basis, i.e., perform outlier analysis on individuals that share the same "identifiers" of sorts. Is there a way to do this in Python?

I am still learning data science so I'm sure this has a term that I am not aware of. Any insight or links to good resources would be greatly appreciated.

$\endgroup$

3 Answers 3

2
$\begingroup$

Yes, the problem of imbalance is indeed genuine while pre processing. There are no hard and fast rules for removing outliers, but generic methodologies (percentile,boxplot,Z-score etc). Like gender, if you take salary of all employess then removing outliers means eliminating all highly paid employees.That will make your model learn more about middle/average salaried employes(Outliers handling). But then if you keep them, they will influence and model will learn less about average salaried employee.(Making data in Log scale can minimise that though:not fully)

The solutions are generally more prone to objective and training we want to give. After pre processing, the imbalance(like gender) in data can be compensated by oversampling or undersampling.(we can get more datapoints:dont worry about that). But be sure before dropping any data that is available !! Taking few columns(similar feature) at a time to process and dealing with them, instead of applying a common operation in groups,can generally works.

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

https://towardsdatascience.com/a-brief-overview-of-outlier-detection-techniques-1e0b2c19e561 These might be helpful.

$\endgroup$
0
$\begingroup$

One solution that should work for most use cases, datasets, and models -- train a model on the data and use the predicted probabilities to obtain a notion of what's normal and what's not normal for all classes. While other answers have mentioned that there is no hard-fast rule to determine what is an outlier, the benefit of this approach is that you can obtain a outlier likelihood score for every example. Then you just choose the cutoff threshold that works for you based on inspection.

A free open-source python package that will do this for you (for any model and any classification dataset as well as unlabeled data) is cleanlab: https://github.com/cleanlab/cleanlab. I am an author on the package (I created it in grad school. Raised funds to hire a team of 10 folks who add new algorithms to it regularly, including for outlier detection).

Outlier detection using feature embeddings

from cleanlab.outlier import OutOfDistribution

ood = OutOfDistribution()

# To get outlier scores for train_data using feature matrix train_feature_embeddings
ood_train_feature_scores = ood.fit_score(features=train_feature_embeddings)

# To get outlier scores for additional test_data using feature matrix test_feature_embeddings
ood_test_feature_scores = ood.score(features=test_feature_embeddings)

Outlier detection using predicted probs from a model

from cleanlab.outlier import OutOfDistribution

ood = OutOfDistribution()

# To get outlier scores for train_data using predicted class probabilities (from a trained classifier) and given class labels
ood_train_predictions_scores = ood.fit_score(pred_probs=train_pred_probs, labels=labels)

# To get outlier scores for additional test_data using predicted class probabilities
ood_test_predictions_scores = ood.score(pred_probs=test_pred_probs)

Tutorial for outlier detection: https://docs.cleanlab.ai/stable/tutorials/outliers.html

ICML Workshop Paper: https://arxiv.org/abs/2207.03061

$\endgroup$
-1
$\begingroup$

No data point should be removed under any circumstances unless you are somehow truly convinced that the data point was acquired in error. The so called "outliers" convey a great amount of information about the system at its boundaries. If your data does contain a point, then how can you justify removing that data point. From an information theory point of view, since "outliers" have such small probability of occurring, they have a large informational content. Removing these points throws away all the information that you were so lucky to observe. The data points that occur with high probability are virtually void of information.

$\endgroup$
3
  • $\begingroup$ Subsampling is a valid technique. "No data point should be removed under any circumstances" doesn't seems valid to me... you are even confirming this in your last sentence: "the data points that occur with high probability are virtually void of information". $\endgroup$ Apr 14, 2022 at 19:14
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Apr 14, 2022 at 19:14
  • $\begingroup$ Subsampling would be valid if and only if you knew the sampling distribution of the original data. Can you subsample a collection of Gaussian distributed samples uniformly? and if you knew the sampling distribution, then there would b no need for this discussion. When I said "the data points that occur with high probability are virtually void of information" please notice the word "virtually" This means they confirm the same information. "outliers" are rare to observe and throwing them away (without definite knowledge that they were obtained in error) is throwing vital information. $\endgroup$
    – ajn
    Apr 18, 2022 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.