Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many ...
0
votes
0answers
16 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
1
vote
1answer
19 views

What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
0
votes
0answers
14 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
2answers
36 views

How to tackle too many outliers in dataset

I boxplot all of my columns with seaborn boxplot in order to know how many outliers that i have, surprisingly there're too many outliers and so i can remove the outliers because i'm afraid with too ...
2
votes
2answers
53 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
1
vote
0answers
19 views

Which outlier detection algorithms give a breakdown of the contribution from each feature?

I am looking for an algorithm that outputs a breakdown of which features contributed the most towards a data point being labelled as an outlier. It can be supervised or unsupervised. At the moment, ...
3
votes
0answers
68 views

Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
0
votes
0answers
10 views

Getting bad predictions for high true values of target variable

I am working on a counterfeit medicine sales prediction regression model. As the relationship between target & response variables is non-linear I used tree based regressors random forests and XGB. ...
2
votes
1answer
34 views

Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?
3
votes
4answers
92 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
1
vote
1answer
62 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
0
votes
1answer
23 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
1
vote
3answers
69 views

Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
0
votes
1answer
17 views

Are cluster feature and micro-cluster good summury statics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
2
votes
1answer
29 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
1
vote
2answers
226 views

how to handle outliers for clustering algorithms?

I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms? Thanks!
1
vote
0answers
14 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
1answer
34 views

What method is recommended after outliers removal?

I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials. I was thinking of ...
0
votes
1answer
127 views

Why labels are not used in outlier detection algorithms?

I read this article from sklearn: https://scikit-learn.org/stable/modules/outlier_detection.html While these algorithms are very useful for outlier detection, I'm surprised to see that they are not ...
0
votes
0answers
10 views

Perform unsupervised anomaly identification with causation in Python?

I have a time series data, which contains information from various sensors measured at every 20 minutes interval. I would like to use information from all these sensors as features to a Deep Learning/...
0
votes
1answer
166 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
1
vote
1answer
28 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
1k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
0
votes
0answers
52 views

dataset shift, covariate shift or sample selection bias in small subsets?

I try to show from which training data size on which machine learning method (CNN, SVM etc.) achieves better performance. For this I would like to use subsets of different sizes from the datasets of ...
1
vote
3answers
58 views

Standard deviation as outlier detection

I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations. When I wanna' use the standard ...
1
vote
2answers
53 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
0
votes
1answer
37 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
19 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
-2
votes
1answer
32 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
1
vote
1answer
185 views

Isolation forest - grouped by

I'm trying to use isolation forest algorithm for outliers detection. Data has 2 columns: id and REV. Below code gives me ...
6
votes
1answer
924 views

In elbow curve how to find the point from where the curve starts to rise?

I am computing a distance metric on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be ...
0
votes
1answer
152 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
1
vote
0answers
36 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
2
votes
0answers
53 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
5
votes
1answer
1k views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
1
vote
2answers
38 views

What is meant by outliers in text data set. How to detect them?

I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes they mentioned that naive-Bayes can affected by the ...
0
votes
1answer
4k views

How to remove outliers using box-plot?

I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)? All the ['AVG'] data is in a single column, I need it ...
0
votes
3answers
55 views

good algorithm for outliers detection

I have 2 independent data sets (1. 300 rows and 2.3000 rows) with 6 months trades observations for 50 traders. In both datasets I have: trader id, stock title, buy/sell volume, date of trade, sector ...
2
votes
1answer
121 views

Number of Nodes in Isolation Forest

I am currently reading this paper on Isolation Forest. At page 3, there is a definition of Isolation Tree and there are a couple of sentences that I don't understand: Given a sample of data X = {...
3
votes
5answers
1k views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
2
votes
5answers
264 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
3
votes
2answers
165 views

How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
0
votes
1answer
35 views

How to distinguish between normal fluctuation and outliers in ARIMA model?

I have a dataset about sales per day of certain products at the ITEM/DAY/STORE level , I've plotted the series and visually examined it for any outliers, volatility, or irregularities. And this is ...
1
vote
1answer
36 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
1
vote
2answers
76 views

Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset): ...
0
votes
2answers
67 views

A Proper Outlier Detection For the Attached Figure

I am wondering how I could detect the red points as outliers using an algorithm (best method for this scenario) not through visualization since it is clear that they are outliers in the figure. ...
1
vote
0answers
41 views

How to find anomalies/outliers in Panel Data?

I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of ...
0
votes
0answers
18 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
1
vote
1answer
84 views

Isolation Forest: simple example

Can some one please explain Isolation Forests more clearly? Everywhere I search, I find the same explanation: Isolation Forest ‘isolates’ observations by randomly selecting a feature and then ...