Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Classification problems - Finding Target variable outliers in data

We have Z-score, IQR etc to identify outliers in data. This could be used to eliminate outliers even in labels. For e.g. if the target variable is a housing price, we could use inter-quartile ranges ...
0
votes
1answer
25 views

Determining extreme outliers from boxplot (by eye) [closed]

I have a normalised dataset with range 0,1. I have created boxplots for every feature in the dataset and need to identify which features have extreme outliers (by looking at their boxplots). However, ...
1
vote
0answers
18 views

What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
4
votes
1answer
380 views

Can a novelty detection model overfit?

Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated ...
2
votes
1answer
21 views

Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
0
votes
1answer
32 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
2
votes
1answer
46 views

Mathematical way of identifying wrong suggestions or outliers

I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person. Eg. Input1 -...
2
votes
1answer
60 views

Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
3
votes
1answer
96 views

Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
0
votes
0answers
26 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
2
votes
2answers
29 views

Boosted tree regression loss function when data has occasionally very large values to predict?

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious ...
0
votes
1answer
48 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
0
votes
2answers
252 views

When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
1
vote
1answer
60 views

Removing outliers with orders of magnitude differences

I have a dataset of virtual currency earn and spend events from a mobile game app. Unfortunately, people cheat in the game to get more currencies. These cheaters use different techniques so its quite ...
1
vote
0answers
19 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
0
votes
0answers
12 views

Is there a way we can narrow down the best range describing a dataset (other than IQR and Min-max)?

I have a dataset of the prices of products for different categories. I want to get the best range which can describe the prices of that category using the Product Data. I have already applied IQR and ...
1
vote
1answer
33 views

How to increase number of outliers in a dataset?

I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering. ...
0
votes
2answers
1k views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
2
votes
0answers
19 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
0
votes
1answer
51 views

Pre-processing - Removing outliers

I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data: It looks like outliers are 4,5,6 and should be removed. Now ...
3
votes
1answer
51 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
0
votes
2answers
39 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
0
votes
0answers
17 views

Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many ...
0
votes
0answers
24 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
1
vote
2answers
34 views

What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
0
votes
0answers
22 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
2answers
432 views

How to tackle too many outliers in dataset

I boxplot all of my columns with seaborn boxplot in order to know how many outliers that i have, surprisingly there're too many outliers and so i can remove the outliers because i'm afraid with too ...
2
votes
2answers
69 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
1
vote
0answers
21 views

Which outlier detection algorithms give a breakdown of the contribution from each feature?

I am looking for an algorithm that outputs a breakdown of which features contributed the most towards a data point being labelled as an outlier. It can be supervised or unsupervised. At the moment, ...
4
votes
3answers
104 views

Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
0
votes
0answers
13 views

Getting bad predictions for high true values of target variable

I am working on a counterfeit medicine sales prediction regression model. As the relationship between target & response variables is non-linear I used tree based regressors random forests and XGB. ...
2
votes
1answer
37 views

Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?
3
votes
4answers
254 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
3
votes
1answer
255 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
0
votes
1answer
27 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
1
vote
3answers
266 views

Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
0
votes
1answer
25 views

Are cluster feature and micro-cluster good summury statics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
2
votes
1answer
36 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
1
vote
2answers
1k views

how to handle outliers for clustering algorithms?

I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms? Thanks!
1
vote
0answers
19 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
1answer
42 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking ...
1
vote
1answer
342 views

Why labels are not used in outlier detection algorithms?

I read this article from sklearn: https://scikit-learn.org/stable/modules/outlier_detection.html While these algorithms are very useful for outlier detection, I'm surprised to see that they are not ...
0
votes
2answers
300 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
1
vote
1answer
34 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
4k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
1
vote
3answers
180 views

Standard deviation as outlier detection

I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations. When I wanna' use the standard ...
1
vote
2answers
62 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
0
votes
1answer
78 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
20 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
-3
votes
1answer
34 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...