17
votes
Accepted
How to remove outliers using box-plot?
Seaborn uses inter-quartile range to detect the outliers. What you need to do is to reproduce the same function in the column you want to drop the outliers. It's quite easy to do in Pandas.
If we ...
- 3,850
13
votes
Accepted
Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?
As a rule of thumb, removing outliers without a good reason to remove outliers rarely does anyone any good. Without a deep and vested understanding of what the possible ranges exist within each ...
- 2,330
13
votes
Accepted
In elbow curve how to find the point from where the curve starts to rise?
TL;DR
Use the two functions from below to get the index of the elbow:
elbow_index = find_elbow(data, get_data_radiant(data))
**Edit:** I put all of the code below ...
- 1,206
11
votes
How to remove outliers using box-plot?
If you need to remove outliers and you need it to work with grouped data, without extra complications, just add showfliers argument as False in the function call. ...
- 263
9
votes
What is the difference between outlier detection and anomaly detection?
(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
- 400
9
votes
Remove or not to remove outliers
This older study Outlier detection and treatment in I/O psychology: A survey of researcher beliefs and an empirical illustration by Orr et al. surveyed a group of psychology researchers about how they ...
- 1,088
8
votes
Outlier detection by unsupervised algorithm: Fraud Detection
First of all, start with a subset until you know what you are doing.
There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it ...
- 7,949
8
votes
Accepted
Difference: Replicator Neural Network vs. Autoencoder
Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
- 1,627
8
votes
Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?
Tophat makes some great points. Another thing to consider is that you removed close to 20 percent of your data by removing the "outliers" which leads me to believe that they really aren't outliers, ...
- 978
7
votes
What is the difference between outlier detection and anomaly detection?
Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data....
- 2,188
7
votes
Accepted
Handling outliers and Null values in Decision tree
Outliers: In decision tree learning, you do splits based on a metric that depends on the proportions of the classes on the left and right leaves after the split (for instance, Giny Impurity). If there ...
- 5,961
6
votes
Handling outliers and Null values in Decision tree
Generally speaking, decision trees are able to handle outliers because their leafs are constructed under metrics which aim to discriminate as much as possible the resulting subsets. Whether you are ...
- 246
6
votes
Accepted
Can a novelty detection model overfit?
Answering your question: yes, depending on the hyperparameters you choose, you could overfit the considered normal data, if you fit your separating hyperplane between normal and novel points being too ...
- 2,674
5
votes
Which algorithms or methods can be used to detect an outlier from this data set?
One way of thinking of outlier detection is that you're creating a predictive model, then you're checking to see if a point falls within the range of predictions. From an information-theoretic point ...
- 151
5
votes
Accepted
Which algorithms or methods can be used to detect an outlier from this data set?
You can use BoxPlot for outlier analysis. I would show you how to do that in Python:
Consider your data as an array:
a = [100, 50, 150, 200, 35, 60 ,50, 20, 500]
...
- 8,156
5
votes
How to identify outliers from a small list of numbers?
An "outlier" is an observation that is so unexpected that we suspect it wasn't valid -- corrupted by noise or something. But what is unexpected? An observation that is highly improbable? But then how ...
- 6,555
5
votes
Accepted
Can GridSearchCV be used for unsupervised learning?
The goal of GridSearchCV is to iterate over (hence search) all possible combinations (hence grid) of hyper parameters and evaluate a model on a cross-validation (...
- 375
4
votes
Tools for automatic anomaly detection on a SQL table?
If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary ...
- 371
4
votes
Accepted
Represent outlier days
One idea would be to plot the daily average power consumption in a bar plot:
For a finer visualization of day-hour peaks, you can try to plot it in a matrix format:
- 76
4
votes
Accepted
Outlier detection for unbalanced classes
you need to distinguish between these cases:
Data Imbalance
Data Imbalance + Very few number of samples (minority class)
Severe Data Imbalance + Very few number of samples (minority class)
20:60 ...
- 1,889
4
votes
Which outlier detection can detect these outliers?
You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor.
...
- 1,103
4
votes
Accepted
Which Outlier Detection Method? Why?
You can justify your choices by using data.
Treat the anomaly detection like a supervised learning problem where the concept is being anomaly.
Then you'll be able to present - for each method - its ...
- 2,613
4
votes
To detect unauthorized access using outlier detection
This question is quite broad. I'll try to set you on the right path, more so than providing a truly complete answer.
Theoretical background
As others have mentioned, the task you're trying to do is ...
- 279
4
votes
Handling outliers and Null values in Decision tree
The answers above are fantastic.
Additionally, what you could do is create a new column and label outliers as 1 (otherwise 0). This is a technique used in Kaggle competitions.
The idea is to make it ...
- 1,027
4
votes
Accepted
4
votes
Which order is correct Feature Selection then Outlier Detection or vice versa?
In majority of the cases feature selection should be done after outlier detection. Outlier detection should be done at the initial stage of data pre-processing while feature extraction / selection can ...
- 758
4
votes
Effect of outliers on Naive Bayes
There are different flavors of Naive Bayes, so the answer depends a bit on the use case.
One potential issue with outliers is that unseen observations can lead to 0 probabilities. For example, ...
oW_♦
- 6,185
4
votes
Isolation Forest: simple example
The idea is that the faster you can isolate a sample the higher the chance it is an outlier/anomaly.
Here is one possible scenario for your example:
...
oW_♦
- 6,185
4
votes
Methods to detect this kind of outliers
I recently had a similar problem (removing abnormal peaks from a time series). That's what I suggest you:
Get the smoothed trend. There are several techniques you can employ, such as various forms of ...
- 5,957
4
votes
how to handle outliers for clustering algorithms?
If you have outliers, the best way is to use a clustering algorithm that can handle them.
For example DBSCAN clustering is robust against outliers when you choose minpts large enough. Don't use k-...
- 7,949
Only top scored, non community-wiki answers of a minimum length are eligible