Questions tagged [outlier]
For questions regarding outliers or unusual points in the data.
215
questions
0
votes
1
answer
41
views
How would normalizing be affected by outliers? And how to avoid it?
I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction.
I'm trying to find the best way to rank all suppliers ...
- 1
0
votes
2
answers
85
views
ML regression for skewed data
I'm trying to build a simple regression model to start with but my Y variable is very skewed to the right. My Y represents the number of views per day for a webpage and all the values are above 0. I ...
- 1
0
votes
1
answer
46
views
Finding outlier for millions of data that can be used cluster-based algorithm
I have millions of data that each have many features. But as long as the value of a feature is not in the acceptable range, that data will be considered an outlier. And I need to find the acceptable ...
- 3
1
vote
1
answer
41
views
Outliers: How to handle data values of future
I am learning ML. I have a dataframe with some features and a target column. For simplicity condiser X has one feature, for eg X ...
- 115
0
votes
1
answer
66
views
How to apply a tree-based model with numerical and categorical values to find outliers
I have a dataset which has a column of prices, a column of dates, and various other columns of numerical and categorical values. I would like to find outlier prices based on all the columns in the ...
0
votes
1
answer
46
views
Dealing with outliers
I'm doing some data analysis on the UCI "Adult Dataset". I have a numerical feature called "hours-per-week" and another feature called "age". These are the only numerical ...
0
votes
1
answer
47
views
Testing RANSAC regression model
I am going to build the model (e.g. multiple linear regression) to predict the appartment cost in my city. First I have to find outliers in training data. For this task RANSAC regression algorithm ...
0
votes
0
answers
27
views
How to detect Novelty from different ranges of target variable?
I've a dataset of multiple categorical columns along with a target column that is continuous. Assume combination of categorical columns has a different range of values of target. Ex
Col1 - col2 - col3 ...
1
vote
1
answer
51
views
Python Library Trend time series multivariate
Our csv contains 36 columns
1 date time column collected every 30 mins
3 variables (count,latency,Totaltime) x 10 Features(user io, serverio ,concurrency ..etc ) Of different data points from the ...
- 11
0
votes
0
answers
14
views
What is the use of validation dataset when doing regression-based outlier detection?
I have a dataset where data are velocity data splitted as:
60%(train - non-anomalous)
20%(validation - 50% of it anomalous)
20%(test - 50% of it anomalous).
From my understanding, when doing outlier ...
- 131
0
votes
0
answers
84
views
Should I remove outlier in features after group based on the result of target variable
I wonder should i remove outliers in case like this? or should i just left it be?
Also if i should remove the outlier, how should i do that? I know how to remove outliers in 1 feature like this
...
0
votes
0
answers
11
views
Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?
I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case?
The system I am building assigns classes (-1, 0, +1) to observations ...
6
votes
2
answers
1k
views
Remove or not to remove outliers
Are there any known academic sources that point towards supporting not removing outliers? Let say if the outlier is a natural occurrence or it has relationship to the value of target variable
- 161
0
votes
0
answers
21
views
Outliers treatment
In a dataset where outliers are present for an IoT device/ manufacturing production line. Is it better not to remove them as these outliers are not measurement errors and could introduce bias in model ...
- 161
2
votes
1
answer
1k
views
Can GridSearchCV be used for unsupervised learning?
im trying to build an outlier detector to find outliers in test data. That data varies a bit (more test channels, longer/shorter testing).
First im applying the train test split because i want to use ...
- 23
0
votes
0
answers
23
views
How to identify outlier images within a set of similar images when no external classification data is available?
I'm trying to automate outlier detection in Python for a computer vision project I'm working on in which many instances of similar objects will be in view of the frame, and I want to verify that all ...
- 1
0
votes
1
answer
70
views
what is the difference between 'object detection' and 'outlier detection' in computer vision?
If I'm looking at drone footage, and I'm looking for tennis courts, then I'm doing object detection.
If I decide that a tennis court is an 'outlier' as opposed to the rest of the landscape, am I now ...
0
votes
0
answers
14
views
Find the most impactfuls parameters multivariate output unsupervised ML
I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
- 1
0
votes
2
answers
329
views
How much percentage of outliers are allowed in a data
I am buliding a machine learning model with logistic regression. I am dealing with blood transfusion data set. In which there is a feature,Total_volume, I found that there is more than 5% of outliers ...
0
votes
2
answers
77
views
Anomaly prediction/forecasting in timeseries?
What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
- 1
0
votes
1
answer
154
views
MAE vs MSE for linear regression
Several articles says that MAE is robust to outliers but MSE is not and MSE can hamper the model if errors are too huge.
My question is that MSE and MAE both are error matrices,our priority is to just ...
- 43
0
votes
1
answer
60
views
How to deal with imbalanced categorical variables in regression tasks?
I want to predict real estate prices using several Machine Learning algorithms. My dataset contains numerical and categorical predictors. I already eliminated the outliers of numerical variables. Now ...
- 1
0
votes
1
answer
308
views
How do outliers and missing values impact these classifiers?
I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class.
I know how to handle finding and ...
1
vote
1
answer
90
views
How variable alpha changes SGDRegressor behavior for outlier?
I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
- 111
1
vote
2
answers
56
views
Given daily sequence of events with only event ID labels (alphanum strings), what algorithms can be used to detect sequences that are outliers?
For example, the data might be something like this:
...
- 11
0
votes
1
answer
84
views
Where can I practice multivariate outlier detection?
Can anyone provide me with a dataset, hopefully on Kaggle, where I can practice my skills in outlier analysis?
I have been studying this topic for quite a while, but I can't find a case study to apply ...
- 121
0
votes
1
answer
95
views
Word2vec to encode medical procedures when using isolation forests
I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data.
Each row of my data represents the group of drugs that each provider has administered in ...
- 271
0
votes
3
answers
801
views
How to remove outliers properly?
I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a ...
- 73
0
votes
1
answer
868
views
Explanation of spectral residual algorithm for outlier detection
I've been reading the paper https://arxiv.org/pdf/1906.03821.pdf for spectral residual outlier detection, but I don't quite understand it. Specifically, in the implementation there are three variables ...
- 11
0
votes
1
answer
60
views
Outlier detection - missing values
I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
- 1
0
votes
1
answer
421
views
How to gps data anomaly detection in python
I have gps format dataset lat, lon. I want to detection anomaly using python.
I tested knn, smv, cof, iforest using pycaret. But i did not.
These colors anomlay because the
angle change is too much
...
- 1
3
votes
2
answers
49
views
How to detect whether an entire series is an outlier relative to others?
I have multiple price series of the same asset as follows.
Visually, it is obvious that series "A" (the flat line) is an outlier, and series "E" (the line with the zig-zag pattern)...
- 131
0
votes
1
answer
42
views
Assign a risk score in records in a dataset
I was wondering, if I have a dataset with categorical and numerical data and labels such as 1 or 0 that shows if a row is anomalous or normal respectively.
Is it possible to create somehow a model ...
0
votes
1
answer
57
views
Can I leave natural outliers in a dataset in training?
Can I leave unedited natural outliers in a dataset (outliers that have not appeared just because of mistyping of mistakes in the data)? Or should I also remove them or change them?
- 33
1
vote
0
answers
114
views
Nonparametric Outlier Detection
Which Nonparametric outlier detection do you suggest to detect outliers (red points) in these plots? I have tested std, IQR, etc., but no good result. It is just one vector including normal and ...
- 443
0
votes
1
answer
85
views
Outlier treatment
I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers.
Remove them (least attractive option)
...
- 1,866
2
votes
1
answer
580
views
Visualizing outliers using T-SNE
I'm trying to visualize outliers in my data using T-SNE and it seems like the outliers appear as three different clusters. The original data has 7 different columns but I chose to plot the outliers on ...
- 21
0
votes
1
answer
928
views
Should outliers be removed only from the target variable or from any variable where they are found?
What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
- 13
0
votes
2
answers
488
views
Removing outliers from a multi-dimensional dataset & Data augmentation
Removing the outliers of a single-dimensional data can be easily done by removing the points that are outside of the IQR range. But how should the process of detecting and removing outliers be done if ...
- 13
1
vote
0
answers
21
views
Methods for comparing datasets for abnormalities
My team at work receives datasets from various companies each quarter, and we're wanting to build some sort of machine learning model that compares the new dataset to our existing data for any ...
- 11
1
vote
1
answer
5k
views
Is standardization/normalization a good way of reducing the impact of outliers when I'm training a machine learning model?
Recently, I have read some papers in which the authors state that they have performed standardization/normalization of the variables for reducing the impact of outliers in the machine learning models ...
- 139
0
votes
0
answers
32
views
Compare standard deviations in different samples?
I have some data which you can group based on different variables.
I know how to test if they have significantly different means. But what the deviation inside the samples?
- 189
1
vote
0
answers
48
views
How can I make live Parallel Coordinates Plot that could show labels for the lines on hover?
I'm using parallel coordinates plot for visualization of my outliers. Is there a way to create them so as to show some information(another column content) about the lie on hovering?
Plotly has ...
- 157
1
vote
2
answers
89
views
How do I determine the top "reason" for anomaly when using Isolation Forests
I am using Isolation Forests for Anomaly Detection.
Say, my set has 10 variables, var1, var2, ..., var10, and I found an anomaly.
Can I rank the 10 variables var1, var2, ..., var10 in such a way I can ...
- 1,971
0
votes
2
answers
1k
views
Anomaly (Outlier) Detection with Isolation Forest too sensitive even with low contamination
I'm trying to use the sklearn implementation of the Isolation Forest algorithm to detect anomalies in my time series data. However, even with a very low contamination parameter (0.0001), it is ...
0
votes
0
answers
124
views
A feature is still right-skewed after log scaling. How should it be normalized for machine learning?
I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
- 811
-1
votes
1
answer
86
views
Anomaly Detection and Removal/Interpolate [closed]
I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
0
votes
1
answer
230
views
Is it a good idea to use parallel coordinates for visualising outliers? [closed]
I tried using parallel coordinates to visualize outliers. Is it fundamentally correct?
- 157
0
votes
0
answers
299
views
How is convex hull method used in outlier detection?
I think the slides are bit unclear on what they want to say. Can someone elaborate this with example.
0
votes
1
answer
66
views
How should a stateless data transformation be applied in regard to train/test split?
I want to apply spatial sign transformation to my data, but unlike other transformations this one is stateless. I am using sklearn and normallly i would first use ...
- 115