Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
2
votes
2answers
39 views

What to replace outliers with? (supermarket transaction data)

I have a transaction dataset from a supermarket. Let's say the average spend is $50. I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend. ...
1
vote
1answer
23 views

The robustness of the Frobenius and L2,1 norms to the outlier

I have a question about the properties of the Frobenius and L$_{2,1}$ norms. Why is the L$_{2,1}$ norm more robust to the outlier than the Frobenius norm? PS: For a matrix $A\in\mathbb{R}^{n\times d}$,...
0
votes
0answers
7 views

Figuring out what's wrong with the box plot. Outliers?

What a 'box plot' of this kind has to say? that basically I have a lot of outliers and I should focus on data in proximity of zero? Thanks!
0
votes
0answers
13 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
0
votes
0answers
14 views

Converting the continuous numerical features into gaussian distribution and how to deal with NaN values after that?

I have a dataset in which there are few continuous numerical features that distribution over them is non gaussian and this means, skewness is nonzero (positive or negative). I read that it is good to ...
2
votes
0answers
44 views

Intuition behind One Class SVM (Scholkopf)

I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space. Why separation from origin with a hyperplane ...
0
votes
0answers
21 views

Classification problems - Finding Target variable outliers in data

We have Z-score, IQR etc to identify outliers in data. This could be used to eliminate outliers even in labels. For e.g. if the target variable is a housing price, we could use inter-quartile ranges ...
0
votes
1answer
28 views

Determining extreme outliers from boxplot (by eye) [closed]

I have a normalised dataset with range 0,1. I have created boxplots for every feature in the dataset and need to identify which features have extreme outliers (by looking at their boxplots). However, ...
1
vote
1answer
39 views

What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
4
votes
1answer
399 views

Can a novelty detection model overfit?

Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated ...
2
votes
1answer
26 views

Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
0
votes
1answer
86 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
2
votes
1answer
50 views

Mathematical way of identifying wrong suggestions or outliers

I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person. Eg. Input1 -...
2
votes
1answer
105 views

Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
3
votes
1answer
164 views

Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
0
votes
0answers
36 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
2
votes
2answers
32 views

Boosted tree regression loss function when data has occasionally very large values to predict?

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious ...
0
votes
1answer
64 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
0
votes
2answers
664 views

When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
1
vote
1answer
81 views

Removing outliers with orders of magnitude differences

I have a dataset of virtual currency earn and spend events from a mobile game app. Unfortunately, people cheat in the game to get more currencies. These cheaters use different techniques so its quite ...
1
vote
0answers
39 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
0
votes
1answer
39 views

How to increase number of outliers in a dataset?

I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering. ...
0
votes
2answers
3k views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
2
votes
0answers
21 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
0
votes
1answer
87 views

Pre-processing - Removing outliers

I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data: It looks like outliers are 4,5,6 and should be removed. Now ...
3
votes
1answer
52 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
1
vote
2answers
58 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
1
vote
0answers
25 views

Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many ...
1
vote
0answers
29 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
1
vote
2answers
38 views

What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
0
votes
0answers
24 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
2answers
831 views

How to tackle too many outliers in dataset

I boxplot all of my columns with seaborn boxplot in order to know how many outliers that i have, surprisingly there're too many outliers and so i can remove the outliers because i'm afraid with too ...
2
votes
2answers
71 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
1
vote
0answers
21 views

Which outlier detection algorithms give a breakdown of the contribution from each feature?

I am looking for an algorithm that outputs a breakdown of which features contributed the most towards a data point being labelled as an outlier. It can be supervised or unsupervised. At the moment, ...
4
votes
3answers
107 views

Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
2
votes
1answer
38 views

Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?
3
votes
4answers
356 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
3
votes
1answer
354 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
0
votes
1answer
28 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
2
votes
3answers
354 views

Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
0
votes
1answer
27 views

Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
2
votes
1answer
72 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
1
vote
2answers
2k views

how to handle outliers for clustering algorithms?

I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms? Thanks!
1
vote
0answers
20 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
1answer
45 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking ...
1
vote
1answer
468 views

Why labels are not used in outlier detection algorithms?

I read this article from sklearn: https://scikit-learn.org/stable/modules/outlier_detection.html While these algorithms are very useful for outlier detection, I'm surprised to see that they are not ...
0
votes
2answers
358 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
1
vote
1answer
37 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
5k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
1
vote
3answers
309 views

Standard deviation as outlier detection

I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations. When I wanna' use the standard ...