# Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

163 questions
Filter by
Sorted by
Tagged with
2answers
73 views

### Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
0answers
7 views

### How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
2answers
304 views

### Replace Values in Vector on Specific Place in R

I want to make $5^{th}$,$10^{th}$,$15^{th}$,$20^{th}$ and $25^{th}$ values of vector an outlier in all xs by using x1 +OT1,...
1answer
50 views

### What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
1answer
39 views

### How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
0answers
27 views

### Problem with Median Absolute Deviation

I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way ...
0answers
46 views

### Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
1answer
36 views

### Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
0answers
12 views

### Robust Gaussian Fit

I have tried to find some literature on robust gaussian fits, all I could find was good old EM gaussian mixtures. The question is : given a mixture of gaussians, find the dominant one around a given ...
3answers
19k views

### How to remove outliers using box-plot?

I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)? All the ['AVG'] data is in a single column, I need it ...
1answer
78 views

### Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
0answers
9 views

### Figuring out what's wrong with the box plot. Outliers?

What a 'box plot' of this kind has to say? that basically I have a lot of outliers and I should focus on data in proximity of zero?
2answers
2k views

### how to handle outliers for clustering algorithms?

I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms?
2answers
258 views

### which outlier detection technique?

I'm new to data science. I have a question on anomaly detection techniques. There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc.. Given a ...
0answers
8 views

### Finding threshold value for right-skewed data (bivariate)

Intuition of the Problem Suppose you have a dataset of two columns, X and Y, and I plot them using a bar plot. The bar plot shows that there are a lot of values in the first two bars on the left, and ...
1answer
50 views

### How to distinguish between normal fluctuation and outliers in ARIMA model?

I have a dataset about sales per day of certain products at the ITEM/DAY/STORE level , I've plotted the series and visually examined it for any outliers, volatility, or irregularities. And this is ...
2answers
127 views

### Notion of cluster centers and cluster comparison in Density Based Algorithms

I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main ...
3answers
2k views

### How to use Autoencoders for outlier detection on images

I have a bunch of images taken from a camera showing a pipe and would like to detect if the pipe is leaking or not. There are very few examples of leaking pipe in the data set. So considering this ...
2answers
70 views

### How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
1answer
39 views

### How to increase number of outliers in a dataset?

I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering. ...
2answers
83 views

### What to replace outliers with? (supermarket transaction data)

I have a transaction dataset from a supermarket. Let's say the average spend is $50. I want to get each customer's average spend and rank them based on where they fall based on this$50 average spend. ...
2answers
38 views

### What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
2answers
69 views

### Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
1answer
125 views

### Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
1answer
50 views

### Mathematical way of identifying wrong suggestions or outliers

I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person. Eg. Input1 -...
1answer
26 views

### The robustness of the Frobenius and L2,1 norms to the outlier

I have a question about the properties of the Frobenius and L$_{2,1}$ norms. Why is the L$_{2,1}$ norm more robust to the outlier than the Frobenius norm? PS: For a matrix $A\in\mathbb{R}^{n\times d}$,...
2answers
799 views

### Which Outlier Detection Method? Why?

For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a ...
1answer
28 views

### Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
1answer
119 views

### Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
1answer
80 views

### Replacing mean by median over batch-size to lessen the impact of outliers

In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less ...
1answer
94 views

### Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
0answers
15 views

### Converting the continuous numerical features into gaussian distribution and how to deal with NaN values after that?

I have a dataset in which there are few continuous numerical features that distribution over them is non gaussian and this means, skewness is nonzero (positive or negative). I read that it is good to ...
0answers
45 views

### Intuition behind One Class SVM (Scholkopf)

I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space. Why separation from origin with a hyperplane ...
1answer
199 views

### Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
3answers
154 views

### What is meant by outliers in text data set. How to detect them?

I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes they mentioned that naive-Bayes can affected by the ...
0answers
26 views

### Classification problems - Finding Target variable outliers in data

We have Z-score, IQR etc to identify outliers in data. This could be used to eliminate outliers even in labels. For e.g. if the target variable is a housing price, we could use inter-quartile ranges ...
1answer
31 views

### Determining extreme outliers from boxplot (by eye) [closed]

I have a normalised dataset with range 0,1. I have created boxplots for every feature in the dataset and need to identify which features have extreme outliers (by looking at their boxplots). However, ...
3answers
6k views

### Difference between Global Outlier and Contextual Outlier?

I am studying "Data Mining: Concepts and Techniques" by Han, Kamber & Pei. In Chapter 12 "Outlier Detection", they have stated that there are 3 types of outliers: Global Outlier - deviates ...
2answers
379 views

### How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
1answer
417 views

### Can a novelty detection model overfit?

Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated ...
1answer
382 views

### Isolation forest - grouped by

I'm trying to use isolation forest algorithm for outliers detection. Data has 2 columns: id and REV. Below code gives me ...
1answer
35 views

### Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
2answers
4k views

### Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
1answer
224 views

### Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
0answers
38 views

### DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
3answers
111 views

### Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
2answers
36 views

### Boosted tree regression loss function when data has occasionally very large values to predict?

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious ...
2answers
932 views

### When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
2answers
387 views

### Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
1answer
100 views

### Removing outliers with orders of magnitude differences

I have a dataset of virtual currency earn and spend events from a mobile game app. Unfortunately, people cheat in the game to get more currencies. These cheaters use different techniques so its quite ...