# Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

162 questions
Filter by
Sorted by
Tagged with
25 views

### Problem with Median Absolute Deviation

I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way ...
45 views

### Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
36 views

### Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
12 views

### Robust Gaussian Fit

I have tried to find some literature on robust gaussian fits, all I could find was good old EM gaussian mixtures. The question is : given a mixture of gaussians, find the dominant one around a given ...
8 views

### Finding threshold value for right-skewed data (bivariate)

Intuition of the Problem Suppose you have a dataset of two columns, X and Y, and I plot them using a bar plot. The bar plot shows that there are a lot of values in the first two bars on the left, and ...
66 views

### What to replace outliers with? (supermarket transaction data)

I have a transaction dataset from a supermarket. Let's say the average spend is $50. I want to get each customer's average spend and rank them based on where they fall based on this$50 average spend. ...
25 views

### The robustness of the Frobenius and L2,1 norms to the outlier

I have a question about the properties of the Frobenius and L$_{2,1}$ norms. Why is the L$_{2,1}$ norm more robust to the outlier than the Frobenius norm? PS: For a matrix $A\in\mathbb{R}^{n\times d}$,...
9 views

### Figuring out what's wrong with the box plot. Outliers?

What a 'box plot' of this kind has to say? that basically I have a lot of outliers and I should focus on data in proximity of zero?
39 views

### How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
15 views

### Converting the continuous numerical features into gaussian distribution and how to deal with NaN values after that?

I have a dataset in which there are few continuous numerical features that distribution over them is non gaussian and this means, skewness is nonzero (positive or negative). I read that it is good to ...
45 views

### Intuition behind One Class SVM (Scholkopf)

I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space. Why separation from origin with a hyperplane ...
26 views

### Classification problems - Finding Target variable outliers in data

We have Z-score, IQR etc to identify outliers in data. This could be used to eliminate outliers even in labels. For e.g. if the target variable is a housing price, we could use inter-quartile ranges ...
31 views

### Determining extreme outliers from boxplot (by eye) [closed]

I have a normalised dataset with range 0,1. I have created boxplots for every feature in the dataset and need to identify which features have extreme outliers (by looking at their boxplots). However, ...
47 views

### What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
414 views

### Can a novelty detection model overfit?

Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated ...
31 views

### Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
117 views

### Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
50 views

### Mathematical way of identifying wrong suggestions or outliers

I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person. Eg. Input1 -...
119 views

### Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
214 views

### Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
37 views

### DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
35 views

### Boosted tree regression loss function when data has occasionally very large values to predict?

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious ...
77 views

### Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
888 views

### When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
98 views

### Removing outliers with orders of magnitude differences

I have a dataset of virtual currency earn and spend events from a mobile game app. Unfortunately, people cheat in the game to get more currencies. These cheaters use different techniques so its quite ...
49 views

### Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
39 views

### How to increase number of outliers in a dataset?

I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering. ...
4k views

### Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
21 views

### Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
110 views

### Pre-processing - Removing outliers

I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data: It looks like outliers are 4,5,6 and should be removed. Now ...
55 views

### Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
66 views

### Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
28 views

### Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many ...
34 views

### Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
38 views

### What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
28 views

### is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
1k views

### How to tackle too many outliers in dataset

I boxplot all of my columns with seaborn boxplot in order to know how many outliers that i have, surprisingly there're too many outliers and so i can remove the outliers because i'm afraid with too ...
72 views

### Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
21 views

### Which outlier detection algorithms give a breakdown of the contribution from each feature?

I am looking for an algorithm that outputs a breakdown of which features contributed the most towards a data point being labelled as an outlier. It can be supervised or unsupervised. At the moment, ...
109 views

### Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
41 views

### Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?
423 views

### Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
433 views

### For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
29 views

### looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
417 views

### Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
27 views

### Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
88 views

### Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
2k views

### how to handle outliers for clustering algorithms?

I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms?