Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

49 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
46 views

Intuition behind One Class SVM (Scholkopf)

I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space. Why separation from origin with a hyperplane ...
2
votes
1answer
50 views

Mathematical way of identifying wrong suggestions or outliers

I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person. Eg. Input1 -...
2
votes
1answer
119 views

Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
2
votes
0answers
21 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
2
votes
0answers
56 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
2
votes
0answers
86 views

Outlier detection: Should the metric used in kNN take into account variance explained by each coordinate?

After applying PCA and working with the reduced dataset, I want to delete the outliers. To do this my idea is to compute the kNN-graph and delete those vertices (points) that have an inner degree of 0....
2
votes
2answers
73 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
1
vote
0answers
27 views

Problem with Median Absolute Deviation

I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way ...
1
vote
0answers
47 views

Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
1
vote
1answer
51 views

What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
1
vote
0answers
50 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
1
vote
2answers
70 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
1
vote
0answers
28 views

Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many ...
1
vote
0answers
36 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
1
vote
2answers
38 views

What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
1
vote
0answers
28 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
1
vote
0answers
21 views

Which outlier detection algorithms give a breakdown of the contribution from each feature?

I am looking for an algorithm that outputs a breakdown of which features contributed the most towards a data point being labelled as an outlier. It can be supervised or unsupervised. At the moment, ...
1
vote
0answers
20 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
1
vote
2answers
70 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
1
vote
0answers
58 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
1
vote
0answers
56 views

How to find anomalies/outliers in Panel Data?

I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of ...
1
vote
1answer
199 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
1
vote
0answers
99 views

What’s more appropriate for outlier detection? Classification or Regression?

I'm using PyTorch and i’m working on a data set where I only care about “outliers” from the norm. The data is comprised of long time series with gaussian distribution, while once in a while there is a ...
1
vote
1answer
94 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
1
vote
0answers
24 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
1
vote
0answers
137 views

Can we use doc2vec to detect outlier documents?

I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task. Or are there any recently evolved, promising algorithms ...
1
vote
0answers
42 views

Taking advatage of the frequency of an outlier

I'm doing outlier detection (Conditional Outliers) on a multivariate time series. The outliers appear every 2 weeks $\pm$ 4 days. How can I incorporate this prior in my models, to reduce the number ...
1
vote
0answers
353 views

Isolation forest results every value -1

I am trying out isolation forest to detect outliers in a specific target column of my dataset. The dataset contains 188 rows of data with 178 rows with the same value for that target column and the ...
1
vote
0answers
365 views

Sensitivity analysis in outlier explanation

I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are ...
1
vote
0answers
261 views

Outlier Detection

I have a dataset which has two class. It has 13 features. They are values which are sent from 13 sensors. Label is True or False. When I use mad outlier detection, when the label is false(really there ...
0
votes
0answers
7 views

How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
0
votes
1answer
36 views

Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
0
votes
0answers
12 views

Robust Gaussian Fit

I have tried to find some literature on robust gaussian fits, all I could find was good old EM gaussian mixtures. The question is : given a mixture of gaussians, find the dominant one around a given ...
0
votes
0answers
8 views

Finding threshold value for right-skewed data (bivariate)

Intuition of the Problem Suppose you have a dataset of two columns, X and Y, and I plot them using a bar plot. The bar plot shows that there are a lot of values in the first two bars on the left, and ...
0
votes
0answers
9 views

Figuring out what's wrong with the box plot. Outliers?

What a 'box plot' of this kind has to say? that basically I have a lot of outliers and I should focus on data in proximity of zero?
0
votes
1answer
39 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
0
votes
0answers
15 views

Converting the continuous numerical features into gaussian distribution and how to deal with NaN values after that?

I have a dataset in which there are few continuous numerical features that distribution over them is non gaussian and this means, skewness is nonzero (positive or negative). I read that it is good to ...
0
votes
0answers
26 views

Classification problems - Finding Target variable outliers in data

We have Z-score, IQR etc to identify outliers in data. This could be used to eliminate outliers even in labels. For e.g. if the target variable is a housing price, we could use inter-quartile ranges ...
0
votes
1answer
129 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
0
votes
0answers
38 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
0
votes
1answer
78 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
0
votes
1answer
39 views

How to increase number of outliers in a dataset?

I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering. ...
0
votes
1answer
28 views

Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
0
votes
0answers
20 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
50 views

How to distinguish between normal fluctuation and outliers in ARIMA model?

I have a dataset about sales per day of certain products at the ITEM/DAY/STORE level , I've plotted the series and visually examined it for any outliers, volatility, or irregularities. And this is ...
0
votes
2answers
258 views

which outlier detection technique?

I'm new to data science. I have a question on anomaly detection techniques. There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc.. Given a ...
0
votes
3answers
2k views

How to use Autoencoders for outlier detection on images

I have a bunch of images taken from a camera showing a pipe and would like to detect if the pipe is leaking or not. There are very few examples of leaking pipe in the data set. So considering this ...
0
votes
2answers
305 views

Replace Values in Vector on Specific Place in R

I want to make $5^{th}$,$10^{th}$,$15^{th}$,$20^{th}$ and $25^{th}$ values of vector an outlier in all xs by using x1 [5]+OT1,...
-1
votes
1answer
132 views

Considering outliers in demand predictions

I have times series data with demand observations during months. I was wondering if, when computing demand predictions, I need to consider the outliers of the observations or not. What is your opinion ...