Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
2
votes
1answer
24 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
1
vote
2answers
41 views

how to handle outliers for clustering algorithms?

I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms? Thanks!
1
vote
0answers
12 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
1answer
27 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking of ...
0
votes
1answer
35 views

Why labels are not used in outlier detection algorithms?

I read this article from sklearn: https://scikit-learn.org/stable/modules/outlier_detection.html While these algorithms are very useful for outlier detection, I'm surprised to see that they are not ...
0
votes
0answers
7 views

Perform unsupervised anomaly identification with causation in Python?

I have a time series data, which contains information from various sensors measured at every 20 minutes interval. I would like to use information from all these sensors as features to a Deep Learning/...
0
votes
1answer
47 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
1
vote
1answer
19 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
270 views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
0
votes
0answers
28 views

dataset shift, covariate shift or sample selection bias in small subsets?

I try to show from which training data size on which machine learning method (CNN, SVM etc.) achieves better performance. For this I would like to use subsets of different sizes from the datasets of ...
1
vote
2answers
40 views

Standard deviation as outlier detection

I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations. When I wanna' use the standard ...
1
vote
2answers
51 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
0
votes
0answers
16 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
18 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
26 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
1
vote
1answer
109 views

Isolation forest - grouped by

I'm trying to use isolation forest algorithm for outliers detection. Data has 2 columns: id and REV. Below code gives me ...
4
votes
1answer
359 views

In elbow curve how to find the point from where the curve starts to rise?

I am computing a distance metric on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be ...
0
votes
1answer
77 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
1
vote
0answers
20 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
2
votes
0answers
51 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
5
votes
1answer
538 views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
1
vote
1answer
24 views

What is meant by outliers in text data set. How to detect them?

I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes they mentioned that naive-Bayes can affected by the ...
0
votes
1answer
901 views

How to remove outliers using box-plot?

I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)? All the ['AVG'] data is in a single column, I need it ...
0
votes
3answers
51 views

good algorithm for outliers detection

I have 2 independent data sets (1. 300 rows and 2.3000 rows) with 6 months trades observations for 50 traders. In both datasets I have: trader id, stock title, buy/sell volume, date of trade, sector ...
2
votes
1answer
108 views

Number of Nodes in Isolation Forest

I am currently reading this paper on Isolation Forest. At page 3, there is a definition of Isolation Tree and there are a couple of sentences that I don't understand: Given a sample of data X = {...
2
votes
5answers
433 views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
2
votes
5answers
185 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
3
votes
2answers
107 views

How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
0
votes
1answer
25 views

How to distinguish between normal fluctuation and outliers in ARIMA model?

I have a dataset about sales per day of certain products at the ITEM/DAY/STORE level , I've plotted the series and visually examined it for any outliers, volatility, or irregularities. And this is ...
0
votes
1answer
30 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
1
vote
2answers
72 views

Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset): ...
0
votes
2answers
63 views

A Proper Outlier Detection For the Attached Figure

I am wondering how I could detect the red points as outliers using an algorithm (best method for this scenario) not through visualization since it is clear that they are outliers in the figure. ...
1
vote
0answers
32 views

How to find anomalies/outliers in Panel Data?

I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of ...
0
votes
0answers
16 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
0
votes
0answers
66 views

Isolation Forest

Can some one please explain Isolation Forests more clearly? Everywhere I search, I find the same explanation: Isolation Forest ‘isolates’ observations by randomly selecting a feature and then ...
2
votes
0answers
41 views

Replacing mean by median over batch-size to lessen the impact of outliers

In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less ...
0
votes
1answer
99 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
2
votes
2answers
83 views

How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

I'm working on an edtech product where some of our traffic lands on webpages about textbooks. Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
6
votes
1answer
102 views

How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I ...
0
votes
2answers
146 views

which outlier detection technique?

I'm new to data science. I have a question on anomaly detection techniques. There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc.. Given a ...
2
votes
1answer
52 views

Gracefully removing observations with outliers in N fields

I have a function. ...
1
vote
1answer
101 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
2
votes
1answer
463 views

Effect of outliers on Naive Bayes

Are Naive Bayes algorithms affected by outliers in the data? Suppose there is a data set, does one need to remove outliers before applying Naive Bayes?
0
votes
3answers
835 views

Why is Local Outlier Factor classified as Unsupervised if it requires training data with no outliers?

In Scikit-Learn, the Local Outlier Factor (LOF) algorithm is defined as an unsupervised anomaly detection method. So then I don't understand why this algorithm requires pre-filtered training data. ...
0
votes
1answer
332 views

Outliers handling

I have a large dataset of >100 columns with nearly all types of data. I want to remove outliers from my dataset for which purpose I've decided to use IQR. Problem is even when I apply quantile of 0.25/...
3
votes
3answers
1k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
1
vote
1answer
61 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
57 views

Finding outliers from multiple files [closed]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
0
votes
1answer
37 views

Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension. For example: ...
0
votes
1answer
101 views

Best way to deal with realistically imbalanced dataset for Regression problem

I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data. NOTE:...