2022 Developer Survey is open! Take survey.

Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
0 votes
1 answer
28 views

How do outliers and missing values impact these classifiers?

I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class. I know how to handle finding and ...
user avatar
1 vote
1 answer
19 views

How variable alpha changes SGDRegressor behavior for outlier?

I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
user avatar
  • 111
1 vote
2 answers
27 views

Given daily sequence of events with only event ID labels (alphanum strings), what algorithms can be used to detect sequences that are outliers?

For example, the data might be something like this: ...
user avatar
  • 11
0 votes
0 answers
15 views

How do I account for outliers in 2-dimensional magnetic field measurements?

I am trying to calculate the magnetic offset of magnetometers. To find the magnetics offset, I spin the magnetometers in a circle and record the magnetic field strength values. This gives me a ...
user avatar
0 votes
0 answers
7 views

What is the impact of an outlier in a dependent variable and independent variable on model performance?

What is the impact of an outlier in a dependent and independent variable on model performance in regression and other machine learning models?. Is outlier in dependent variable more impactful than in ...
user avatar
  • 109
0 votes
1 answer
24 views

Where can I practice multivariate outlier detection?

Can anyone provide me with a dataset, hopefully on Kaggle, where I can practice my skills in outlier analysis? I have been studying this topic for quite a while, but I can't find a case study to apply ...
user avatar
0 votes
0 answers
18 views

Outlier removal

I'm trying to predict a binary variable. One of the explanatory variables has some outliers on it. But if I analyze this explanatory variable on its own, I get a set of outliers. Now, when I analyze ...
user avatar
0 votes
1 answer
36 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
user avatar
  • 231
0 votes
0 answers
6 views

Propagating -infs in pytorch and outliers in general

I am using a loss which requires sampling from probability distributions to do monte carlo integration with. Sometimes legitimate training data can throw -inf/NaN. ...
user avatar
0 votes
2 answers
113 views

How to remove outliers properly?

I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a ...
user avatar
  • 73
0 votes
1 answer
83 views

Explanation of spectral residual algorithm for outlier detection

I've been reading the paper https://arxiv.org/pdf/1906.03821.pdf for spectral residual outlier detection, but I don't quite understand it. Specifically, in the implementation there are three variables ...
user avatar
  • 11
0 votes
1 answer
31 views

Outlier detection - missing values

I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
user avatar
  • 1
0 votes
0 answers
120 views

SHAP Kernel explainer for ensemble model

I am currently working on a project involving an unsupervised outlier detection ensemble model. However I am getting stuck by an error passed by the shap.KernelExplainer: "The passed model is not ...
user avatar
0 votes
1 answer
92 views

How to gps data anomaly detection in python

I have gps format dataset lat, lon. I want to detection anomaly using python. I tested knn, smv, cof, iforest using pycaret. But i did not. These colors anomlay because the angle change is too much ...
user avatar
  • 1
4 votes
2 answers
41 views

How to detect whether an entire series is an outlier relative to others?

I have multiple price series of the same asset as follows. Visually, it is obvious that series "A" (the flat line) is an outlier, and series "E" (the line with the zig-zag pattern)...
user avatar
  • 141
0 votes
1 answer
31 views

Assign a risk score in records in a dataset

I was wondering, if I have a dataset with categorical and numerical data and labels such as 1 or 0 that shows if a row is anomalous or normal respectively. Is it possible to create somehow a model ...
user avatar
0 votes
1 answer
27 views

Can I leave natural outliers in a dataset in training?

Can I leave unedited natural outliers in a dataset (outliers that have not appeared just because of mistyping of mistakes in the data)? Or should I also remove them or change them?
user avatar
  • 33
1 vote
0 answers
53 views

Nonparametric Outlier Detection

Which Nonparametric outlier detection do you suggest to detect outliers (red points) in these plots? I have tested std, IQR, etc., but no good result. It is just one vector including normal and ...
user avatar
  • 443
0 votes
1 answer
32 views

Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
user avatar
  • 1,241
2 votes
1 answer
76 views

Visualizing outliers using T-SNE

I'm trying to visualize outliers in my data using T-SNE and it seems like the outliers appear as three different clusters. The original data has 7 different columns but I chose to plot the outliers on ...
user avatar
0 votes
1 answer
115 views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
user avatar
0 votes
0 answers
19 views

Regression followed by thresholding to predict rare events

I have a multi-variate time series for which I am performing forecasting by regression. My aim is to forecast extreme values in this time series (rare events). On the one hand I have a regression ...
user avatar
  • 121
0 votes
2 answers
73 views

Removing outliers from a multi-dimensional dataset & Data augmentation

Removing the outliers of a single-dimensional data can be easily done by removing the points that are outside of the IQR range. But how should the process of detecting and removing outliers be done if ...
user avatar
1 vote
0 answers
18 views

Methods for comparing datasets for abnormalities

My team at work receives datasets from various companies each quarter, and we're wanting to build some sort of machine learning model that compares the new dataset to our existing data for any ...
user avatar
1 vote
1 answer
306 views

Is standardization/normalization a good way of reducing the impact of outliers when I'm training a machine learning model?

Recently, I have read some papers in which the authors state that they have performed standardization/normalization of the variables for reducing the impact of outliers in the machine learning models ...
user avatar
  • 127
0 votes
0 answers
30 views

Compare standard deviations in different samples?

I have some data which you can group based on different variables. I know how to test if they have significantly different means. But what the deviation inside the samples?
user avatar
1 vote
0 answers
22 views

How can I make live Parallel Coordinates Plot that could show labels for the lines on hover?

I'm using parallel coordinates plot for visualization of my outliers. Is there a way to create them so as to show some information(another column content) about the lie on hovering? Plotly has ...
user avatar
1 vote
2 answers
55 views

How do I determine the top "reason" for anomaly when using Isolation Forests

I am using Isolation Forests for Anomaly Detection. Say, my set has 10 variables, var1, var2, ..., var10, and I found an anomaly. Can I rank the 10 variables var1, var2, ..., var10 in such a way I can ...
user avatar
  • 1,851
0 votes
2 answers
299 views

Anomaly (Outlier) Detection with Isolation Forest too sensitive even with low contamination

I'm trying to use the sklearn implementation of the Isolation Forest algorithm to detect anomalies in my time series data. However, even with a very low contamination parameter (0.0001), it is ...
user avatar
0 votes
0 answers
20 views

Evaluating unsupervised Oulier Detection Models

I'm trying to find ways to evaluate unsupervised outlier detection models like Isolation Forest, One-class SVM, COPOD etc. I found this paper How to Evaluate the Quality of Unsupervised Anomaly ...
user avatar
0 votes
0 answers
17 views

Outliers capping is leading to generation of new duplicates during data pre-processing

So as the title suggest, I removed duplicates which were around 5% to the data but after outliers capping, new duplicates got generated in huge amount (~8%) so what should I do in this case? I'm going ...
user avatar
0 votes
0 answers
61 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
user avatar
  • 791
-1 votes
1 answer
54 views

Anomaly Detection and Removal/Interpolate [closed]

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
user avatar
0 votes
1 answer
126 views

Is it a good idea to use parallel coordinates for visualising outliers? [closed]

I tried using parallel coordinates to visualize outliers. Is it fundamentally correct?
user avatar
0 votes
0 answers
190 views

Novelty prediction Using DBSCAN on "unseen data"

I am trying to build an unsupervised learning model, which will be able to predict outliers on "unseen data." The algorithm I chose is DBSCAN (Density-based spatial clustering of ...
user avatar
0 votes
0 answers
63 views

How is convex hull method used in outlier detection?

I think the slides are bit unclear on what they want to say. Can someone elaborate this with example.
user avatar
0 votes
1 answer
34 views

How should a stateless data transformation be applied in regard to train/test split?

I want to apply spatial sign transformation to my data, but unlike other transformations this one is stateless. I am using sklearn and normallly i would first use ...
user avatar
  • 115
0 votes
1 answer
42 views

Distance between any two points after DBSCAN

DBSCAN is a clustering model which is robust to detect the outliers also. A parameter $\epsilon$ i.e. radius is an input of the algorithm, a point is said to be outlier if it's circle with radius $\...
user avatar
0 votes
0 answers
26 views

Data preprocessing - Time Series data resets its values - Detection & Correction

1. Summarize the problem I currently trying to work with time series data from sensors which has some problems regarding resetting it values. For example some cumulative values gets reset and don't ...
user avatar
0 votes
2 answers
48 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
user avatar
1 vote
1 answer
77 views

What if outliers still exist after variable transformation?

I have a variable with a skewed distribution. I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
user avatar
  • 45
1 vote
0 answers
57 views

Model Tree M5 - Robustness to Data Quality Issues

I am currently investigating the M5 tree algorithm by Quinlan(1992) link here: https://sci2s.ugr.es/keel/pdf/algorithm/congreso/1992-Quinlan-AI.pdf An example of a linear regression model of the ...
user avatar
-1 votes
1 answer
56 views

How to aggregate data inserted by users to avoid outliers?

I'm developing a new application based on machine learning. In this application users can insert new data to improve the prediction system. As you may guess, users could insert data that doesn't make ...
user avatar
  • 1
0 votes
1 answer
27 views

Clustering method for 2-D data that self-detects number of clusters and takes care of outliers

Assuming I have data that looks something like that: I'm looking for a method or algorithm that can perform the clustering (e.g. as shown in the picture), that automatically determines the optimal ...
user avatar
  • 101
1 vote
1 answer
906 views

How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
user avatar
  • 369
1 vote
1 answer
93 views

Problem with Median Absolute Deviation

I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way ...
user avatar
  • 71
1 vote
0 answers
187 views

Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
user avatar
  • 71
0 votes
2 answers
67 views

Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
user avatar
2 votes
2 answers
555 views

What to replace outliers with? (supermarket transaction data)

I have a transaction dataset from a supermarket. Let's say the average spend is $50. I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend. ...
user avatar
  • 123
1 vote
1 answer
44 views

The robustness of the Frobenius and L2,1 norms to the outlier

I have a question about the properties of the Frobenius and L$_{2,1}$ norms. Why is the L$_{2,1}$ norm more robust to the outlier than the Frobenius norm? PS: For a matrix $A\in\mathbb{R}^{n\times d}$,...
user avatar
  • 113

1
2 3 4 5