Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
10
votes
4answers
758 views

Gas consumption outliers detection - Neural network project. Bad results

I tried to detect outliers in the energy gas consumption of some dutch buildings, building a neural network model. I have very bad results, but I can't find the reason. I am not an expert so I would ...
10
votes
2answers
2k views

Scalable Outlier/Anomaly Detection

I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms ...
10
votes
2answers
5k views

Tools for automatic anomaly detection on a SQL table?

I have a large SQL table that is essentially a log. The data is pretty complex and I'm trying to find some way to identify anomalies without me understanding all the data. I've found lots of tools for ...
8
votes
3answers
6k views

What is the difference between outlier detection and anomaly detection?

I would like to know the difference in terms of applications (e.g. which one is credit card fraud detection?) and in terms of used techniques. Example papers which define the task would be welcome.
8
votes
3answers
800 views

Which algorithms or methods can be used to detect an outlier from this data set?

Suppose I have a data set : Amount of money (100, 50, 150, 200, 35, 60 ,50, 20, 500). I have Googled the web looking for techniques that can be used to find a ...
8
votes
1answer
6k views

Difference: Replicator Neural Network vs. Autoencoder

I'm currently studying papers about outlier detection using RNN's (Replicator Neural Networks) and wonder what is the particular difference to Autoencoders? RNN's seem to be treaded for many as the ...
7
votes
3answers
998 views

Which outlier detection can detect these outliers?

I have a vector and want to detect outliers in it. The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal. ...
7
votes
1answer
442 views

How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I ...
6
votes
1answer
2k views

In elbow curve how to find the point from where the curve starts to rise?

I am computing a distance metric on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be ...
6
votes
2answers
901 views

Anomaly detection in time series

The use case : Everyday, we have metrics that are established daily to check that our systems are doing fine. From time to times, bugs occur in the workflow building these metrics, and I have to ...
5
votes
3answers
12k views

How to remove outliers using box-plot?

I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)? All the ['AVG'] data is in a single column, I need it ...
5
votes
3answers
1k views

To detect unauthorized access using outlier detection

I am working on project where my task is to find unauthorized access using any machine learning technique. Let me clear my problem definition. UserA access website using chrome browser from windows ...
5
votes
2answers
1k views

Effect of outliers on Naive Bayes

Are Naive Bayes algorithms affected by outliers in the data? Suppose there is a data set, does one need to remove outliers before applying Naive Bayes?
5
votes
2answers
292 views

How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
5
votes
3answers
516 views

Do you apply outlier detection of numerical data in practical applications?

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data. Do you apply outlier detection in your ...
5
votes
1answer
2k views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
4
votes
3answers
5k views

Handling outliers and Null values in Decision tree

Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example? (What if I have a features ranging from 0 to 9 but there is an ...
4
votes
1answer
380 views

Can a novelty detection model overfit?

Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated ...
4
votes
2answers
75 views

Represent outlier days

I have hourly power consumption data for 30 days. On representing, each day data using a separate line, I get a plot as I want to highlight the days with abnormally high consumption (in other words, ...
4
votes
1answer
5k views

Multivariate outlier detection with isolation forest..How to detect most effective features?

I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/...
4
votes
3answers
104 views

Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
3
votes
3answers
2k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
3
votes
2answers
4k views

Outlier detection by unsupervised algorithm: Fraud Detection

I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset. I have created around 5 features (All continuous data, with ...
3
votes
2answers
778 views

Which Outlier Detection Method? Why?

For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a ...
3
votes
2answers
2k views

Outlier detection for unbalanced classes

I have to make a predictive model for predicting a boolean Won/Lost variable based on some other numeric data; and further find out the features of observations that have 'Won'. However, the number ...
3
votes
1answer
647 views

How to identify outliers from a small list of numbers?

I want to identify outliers from a very small group of numbers. How shall I do that? For example, for the group of numbers: -0.4, 0.4, 52.1, actually 52.1 is an outlier. I've tried using ...
3
votes
4answers
256 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
3
votes
1answer
51 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
3
votes
5answers
3k views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
3
votes
2answers
4k views

When to remove outlier in preparing features for machine learning algorithm

I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, ...
3
votes
1answer
96 views

Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
3
votes
1answer
256 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
3
votes
2answers
1k views

Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the ...
3
votes
2answers
756 views

How to increase the weight when it comes to outlier detection

Let's say we have feature A, B, C, D, E to represent one observation in an outlier detection model. We are using scikit-learn outlier detection in our case. AFAIK, if we normalize all the features, ...
3
votes
1answer
4k views

Anomaly Detection In Univariate Time Series Data Using ARIMA In Python With Updating

I have trained an ARIMA model on some 15 minute incremented time series data by using the statsmodels library. I would like to determine how anomalous the next 15 minute increment's data I observe is. ...
3
votes
2answers
521 views

How can I identify and remove outliers in R

I am performing regression analysis on prices of product that we have purchased, based on size and other attributes. However there are often buys in odd circumstances which factor into the price, ...
2
votes
5answers
489 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
2
votes
2answers
257 views

General methods outlier detection

What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of ...
2
votes
2answers
29 views

Boosted tree regression loss function when data has occasionally very large values to predict?

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious ...
2
votes
1answer
36 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
2
votes
3answers
64 views

What is meant by outliers in text data set. How to detect them?

I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes they mentioned that naive-Bayes can affected by the ...
2
votes
1answer
142 views

Number of Nodes in Isolation Forest

I am currently reading this paper on Isolation Forest. At page 3, there is a definition of Isolation Tree and there are a couple of sentences that I don't understand: Given a sample of data X = {...
2
votes
1answer
97 views

Some algorithms and approach for identification of specific patterns?

I am working on a hobby project where I have a data related to financial trades(such as stock trades). Now I want to analyse this data and extract possible ...
2
votes
2answers
112 views

How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

I'm working on an edtech product where some of our traffic lands on webpages about textbooks. Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
2
votes
3answers
573 views

Why replacing null values with outliers?

I have been watching a tutorial on stock price prediction with multivariate linear regression and the tutor replaces missing value data, NaN, with the outlier -99999. Why and how do replacements like ...
2
votes
2answers
724 views

Outlier detection and removal using scatter-plots and histograms

Suppose, we use the following code to generate scatter plots, ...
2
votes
1answer
380 views

Remove Outliers - Market Basket Analysis

I'm having some thoughts on whether I should remove the outliers. I'm trying to find the tags that are commonly used together. Imagine that I have the following dataset. The first column is the Tag_ID ...
2
votes
2answers
2k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...
2
votes
1answer
21 views

Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
2
votes
1answer
37 views

Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?