Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
0
votes
0answers
3 views

Perform unsupervised anomaly identification with causation in Python?

I have a time series data, which contains information from various sensors measured at every 20 minutes interval. I would like to use information from all these sensors as features to a Deep Learning/...
0
votes
1answer
21 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
-1
votes
0answers
13 views

Outliers detection in high dimensional space [closed]

I'm trying to use gaussian mixture for outliers detection on high dimensional space. I don't know how !! cluster scatter plot with two first principales components
0
votes
0answers
11 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
47 views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
0
votes
0answers
20 views

dataset shift, covariate shift or sample selection bias in small subsets?

I try to show from which training data size on which machine learning method (CNN, SVM etc.) achieves better performance. For this I would like to use subsets of different sizes from the datasets of ...
1
vote
2answers
38 views

Standard deviation as outlier detection

I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations. When I wanna' use the standard ...
1
vote
2answers
47 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
0
votes
0answers
14 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
18 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
26 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
1
vote
1answer
64 views

Isolation forest - grouped by

I'm trying to use isolation forest algorithm for outliers detection. Data has 2 columns: id and REV. Below code gives me ...
4
votes
1answer
206 views

In elbow curve how to find the point from where the curve starts to rise?

I am computing a distance on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be discarded. ...
0
votes
1answer
60 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
1
vote
0answers
17 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
2
votes
0answers
48 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
4
votes
1answer
372 views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
1
vote
1answer
22 views

What is meant by outliers in text data set. How to detect them?

I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes they mentioned that naive-Bayes can affected by the ...
0
votes
1answer
282 views

How to remove outliers using box-plot?

I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)? All the ['AVG'] data is in a single column, I need it ...
0
votes
2answers
40 views

good algorithm for outliers detection

I have 2 independent data sets (1. 300 rows and 2.3000 rows) with 6 months trades observations for 50 traders. In both datasets I have: trader id, stock title, buy/sell volume, date of trade, sector ...
2
votes
1answer
101 views

Number of Nodes in Isolation Forest

I am currently reading this paper on Isolation Forest. At page 3, there is a definition of Isolation Tree and there are a couple of sentences that I don't understand: Given a sample of data X = {...
2
votes
5answers
184 views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
2
votes
5answers
166 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
3
votes
2answers
86 views

How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
0
votes
1answer
24 views

How to distinguish between normal fluctuation and outliers in ARIMA model?

I have a dataset about sales per day of certain products at the ITEM/DAY/STORE level , I've plotted the series and visually examined it for any outliers, volatility, or irregularities. And this is ...
0
votes
1answer
29 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
1
vote
2answers
71 views

Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset): ...
0
votes
2answers
63 views

A Proper Outlier Detection For the Attached Figure

I am wondering how I could detect the red points as outliers using an algorithm (best method for this scenario) not through visualization since it is clear that they are outliers in the figure. ...
1
vote
0answers
30 views

How to find anomalies/outliers in Panel Data?

I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of ...
0
votes
0answers
14 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
0
votes
0answers
61 views

Isolation Forest

Can some one please explain Isolation Forests more clearly? Everywhere I search, I find the same explanation: Isolation Forest ‘isolates’ observations by randomly selecting a feature and then ...
2
votes
0answers
38 views

Replacing mean by median over batch-size to lessen the impact of outliers

In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less ...
0
votes
1answer
90 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
2
votes
2answers
81 views

How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

I'm working on an edtech product where some of our traffic lands on webpages about textbooks. Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
6
votes
1answer
92 views

How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I ...
0
votes
2answers
129 views

which outlier detection technique?

I'm new to data science. I have a question on anomaly detection techniques. There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc.. Given a ...
2
votes
1answer
51 views

Gracefully removing observations with outliers in N fields

I have a function. ...
1
vote
1answer
94 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
2
votes
1answer
389 views

Effect of outliers on Naive Bayes

Are Naive Bayes algorithms affected by outliers in the data? Suppose there is a data set, does one need to remove outliers before applying Naive Bayes?
0
votes
3answers
710 views

Why is Local Outlier Factor classified as Unsupervised if it requires training data with no outliers?

In Scikit-Learn, the Local Outlier Factor (LOF) algorithm is defined as an unsupervised anomaly detection method. So then I don't understand why this algorithm requires pre-filtered training data. ...
0
votes
1answer
274 views

Outliers handling

I have a large dataset of >100 columns with nearly all types of data. I want to remove outliers from my dataset for which purpose I've decided to use IQR. Problem is even when I apply quantile of 0.25/...
3
votes
3answers
1k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
1
vote
1answer
59 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
57 views

Finding outliers from multiple files [closed]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
0
votes
1answer
37 views

Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension. For example: ...
0
votes
1answer
96 views

Best way to deal with realistically imbalanced dataset for Regression problem

I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data. NOTE:...
1
vote
0answers
22 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
2
votes
1answer
614 views

Collinearity and Outlier Removal

I am playing with a credit fraud detection dataset at Kaggle. An imbalanced dataset with about 0.1% of fraud transaction. The features are 28 PCs out from a PCA exercise done by the data publisher + ...
1
vote
1answer
315 views

Which order is correct Feature Selection then Outlier Detection or vice versa?

which of these orders is correct? First (Feature Selection) Second (Outlier Detection) or First (Outlier Detection) Second (Feature Selection)
0
votes
3answers
859 views

How to use Autoencoders for outlier detection on images

I have a bunch of images taken from a camera showing a pipe and would like to detect if the pipe is leaking or not. There are very few examples of leaking pipe in the data set. So considering this ...