Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

0
votes
0answers
11 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
0
votes
1answer
24 views

Isolation Forest

Can some one please explain Isolation Forests more clearly? Everywhere I search, I find the same explanation: Isolation Forest ‘isolates’ observations by randomly selecting a feature and then ...
2
votes
0answers
24 views

Replacing mean by median over batch-size to lessen the impact of outliers

In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less ...
0
votes
1answer
37 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
1
vote
1answer
57 views

How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

I'm working on an edtech product where some of our traffic lands on webpages about textbooks. Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
1
vote
1answer
34 views

How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I ...
0
votes
1answer
18 views

which outlier detection technique?

I'm new to data science. I have a query on anomaly detection techniques. There are several anomaly detection techniques Like statistical, density based, depth based , clustering etc.. Given a ...
2
votes
1answer
39 views
1
vote
1answer
46 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
2
votes
1answer
109 views

Effect of outliers on Naive Bayes

Are Naive Bayes algorithms affected by outliers in the data? Suppose there is a data set, does one need to remove outliers before applying Naive Bayes?
0
votes
2answers
106 views

Why is Local Outlier Factor classified as Unsupervised if it requires training data with no outliers?

In Scikit-Learn, the Local Outlier Factor (LOF) algorithm is defined as an unsupervised anomaly detection method. So then I don't understand why this algorithm requires pre-filtered training data. ...
0
votes
1answer
41 views

Outliers handling

I have a large dataset of >100 columns with nearly all types of data. I want to remove outliers from my dataset for which purpose I've decided to use IQR. Problem is even when I apply quantile of 0.25/...
3
votes
3answers
685 views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
1
vote
1answer
38 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
50 views

Finding outliers from multiple files [closed]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
0
votes
1answer
28 views

Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension. For example: ...
0
votes
1answer
65 views

Best way to deal with realistically imbalanced dataset for Regression problem

I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data. NOTE:...
1
vote
0answers
15 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
2
votes
1answer
177 views

Collinearity and Outlier Removal

I am playing with a credit fraud detection dataset at Kaggle. An imbalanced dataset with about 0.1% of fraud transaction. The features are 28 PCs out from a PCA exercise done by the data publisher + ...
0
votes
1answer
127 views

Which order is correct Feature Selection then Outlier Detection or vice versa?

which of these orders is correct? First (Feature Selection) Second (Outlier Detection) or First (Outlier Detection) Second (Feature Selection)
0
votes
2answers
430 views

How to use Autoencoders for outlier detection on images

I have a bunch of images taken from a camera showing a pipe and would like to detect if the pipe is leaking or not. There are very few examples of leaking pipe in the data set. So considering this ...
0
votes
1answer
39 views

Is there an upper bound for k in nearest neighbors-based methods?

When applying a nearest neighbors-based method to a data of, for instance, 2000 points, what is the largest number of neighbors that can be considered ? I am using a nearest neighbors method in an ...
0
votes
1answer
29 views

How to make multiple regression perform better for outliers? (without reducing effect of them)

I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values. So I ...
0
votes
2answers
51 views

Replace Values in Vector on Specific Place in R

I want to make $5^{th}$,$10^{th}$,$15^{th}$,$20^{th}$ and $25^{th}$ values of vector an outlier in all xs by using x1 [5]+OT1,...
2
votes
1answer
73 views

Probabilistic Outlier Detection (edited + clarified)

Measured data $D \in \mathbb{R}^3$, every $d^i \in D$ is $d^i_{(x)}$, where the $x=[x_1, x_2]$. Simply said, the measured data are function of $x$. It is known, the dependency is linear, such as: $$...
1
vote
0answers
62 views

Can we use doc2vec to detect outlier documents?

I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task. Or are there any recently evolved, promising algorithms ...
3
votes
1answer
23 views

Should we identify outliers from population prior to taking sample?

I am revising undergrad statistics course via this course, where i am learning techniques to pull out sample from population. While ensuring that sample is a decent representative of the population, ...
1
vote
0answers
35 views

Taking advatage of the frequency of an outlier

I'm doing outlier detection (Conditional Outliers) on a multivariate time series. The outliers appear every 2 weeks $\pm$ 4 days. How can I incorporate this prior in my models, to reduce the number ...
2
votes
3answers
238 views

Why replacing null values with outliers?

I have been watching a tutorial on stock price prediction with multivariate linear regression and the tutor replaces missing value data, NaN, with the outlier -99999. Why and how do replacements like ...
1
vote
1answer
3k views

Remove Local Outliers from Dataframe using pandas

Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....
1
vote
1answer
385 views

LOF gives same number of outliers irrespective of parameters

I am running lof algorithm for around 100k 2d points. Each time, I run the lof algorithm with different n_neighbours parameter, I get the same number of points as outliers. It's always 10% of the ...
0
votes
2answers
45 views

Unsupervised Clustering of 6-d vectors

I have 17k vectors, each having 6 points, I want to cluster the vectors based on nature of points, eg. linearly increasing into one cluster, convex in another cluster, concave in another, decreasing ...
0
votes
1answer
2k views

Implement Sliding Window Algorithm to Detect Spikes

How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.
2
votes
2answers
821 views

Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the ...
1
vote
2answers
480 views

Using local outlier factor score to detect outliers at run time

I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my ...
3
votes
3answers
2k views

Handling outliers and Null values in Decision tree

Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example? (What if I have a features ranging from 0 to 9 but there is an ...
-1
votes
1answer
52 views

Understanding a Box plot

I have a dataset and I have tried plotting it in a box plot after loading the csv in usings pandas. The plot itself is given below. I need to understand why the x1 ...
1
vote
1answer
88 views

Principal Component Analysis and abnormal data

I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow ...
4
votes
1answer
2k views

Multivariate outlier detection with isolation forest..How to detect most effective features?

I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/...
2
votes
2answers
123 views

General methods outlier detection

What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of ...
1
vote
0answers
152 views

Is there any tool to remove outlier from dataset?

I have a multivariate data set and the target variable is nominal. I want to remove outliers from my dataset. Is there any tool that can help me or I have to write the code by myself?
0
votes
1answer
32 views

outlier detection in time serie without using windows [closed]

I would like to know if it's possible to detect outliers in a time-serie with an outlier score computed given the whole dataset and not given windows
1
vote
0answers
263 views

Isolation forest results every value -1

I am trying out isolation forest to detect outliers in a specific target column of my dataset. The dataset contains 188 rows of data with 178 rows with the same value for that target column and the ...
1
vote
2answers
2k views

When to remove outlier in preparing features for machine learning algorithm

I have a numeric variable (price) and it has long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, ...
2
votes
1answer
189 views

An outlier detection for this data

I have a vector and want to detect outliers in it. I need an outlier detection method (a non-parametric method) which can detect red points as outliers. Edit: I have a lot of vectors like this. The ...
5
votes
3answers
1k views

To detect unauthorized access using outlier detection

I am working on project where my task is to find unauthorized access using any machine learning technique. Let me clear my problem definition. UserA access website using chrome browser from windows ...
0
votes
1answer
51 views

Differentiating between normal and attack clusters

In the below figure, we plotted some data of sensors in normal condition and under attack(outlier): 1. Green points are normal samples in the training dataset. 2. Cyan and red points are normal ...
1
vote
1answer
488 views

Isolation Forest height limit absent in SkLearn implementation

In the original publication of the Isolation Forest algorithm, the authors mention a height limit parameter to control the granularity of the algorithm. I did not find that explicit parameter on the ...
1
vote
4answers
290 views

How to detect anomalous points

As is clear from the figure, the blue points, which don't follow the trend, are anomalous points. I'm wondering about the best non-parametric method to detect those points. I have tested some ...
1
vote
0answers
165 views

Sensitivity analysis in outlier explanation

I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are ...