Skip to main content

Questions tagged [outlier]

For questions regarding outliers or unusual points in the data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

How important is outlier clearing

When I'm doing data pre-processing , I always handle outlier, whether its using mean, median , and sometimes deleting it. But i realize that sometimes handling it just makes the accuracy lower, so i ...
Razark's user avatar
  • 1
0 votes
0 answers
67 views

Confused with Isolation Forest

Let say, I have the anomaly detection (unsupervised learning) dataset with 10 observations (two features). The datasets is like below: After executing the model, following are the results (anomalies ...
Bits's user avatar
  • 131
0 votes
1 answer
19 views

How to identify outliers on a box and whisker plot that seems to be compressed?

I have plotted box plots for the features of an ML problem, to identify outliers. I have scaled the data using a MinMaxScaler so that the scaled data is in the range [0,1]. For some columns, the two ...
san's user avatar
  • 1
0 votes
1 answer
24 views

can we use tanh activation function to detect outliers?

Can we use tanh activation function to detect outliers ? Does my image below true for dataset outliers (after training model with tanh activation function) ?
user3668129's user avatar
1 vote
0 answers
24 views

Outlier detection with elliptic envelope - unexpected error

I am trying to detect outliers with sklearn.covariance.EllipticEnvelope for a single variable, but it throws an unexpected error. Here is an example the reproduces ...
Maya's user avatar
  • 11
0 votes
0 answers
16 views

Outlier Detection for non linear data for regression

I'm grappling with two parameters exhibiting an 82% correlation, showcasing a non-linear relationship and continuous fluctuations. Despite attempting LocalOutlierFactor in scikit-learn, outlier ...
RAJESH KOYI's user avatar
0 votes
1 answer
105 views

Min-Max Scaling more sensitive to outliers than 'Simple Feature Scaling'?

I am confused as to the pros and cons of two different approaches to normalization: Min-Max Scaling, and what the lecturer in the course I am taking refers to as 'Simple Feature Scaling'. The latter ...
Chris Bedford's user avatar
0 votes
1 answer
230 views

Outlier Handing when most value is 0

Just a question, i know that when we plot against the distribution of numerical data, those who fall outside of the boxplot (diamond shape point) are considered outlier. However, i met a case where ...
Razark's user avatar
  • 1
0 votes
1 answer
22 views

Univariate anomaly / outlier detection

I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection. I have a dataset of around 60K data points. Each data point is part ...
EyalG's user avatar
  • 1
0 votes
0 answers
15 views

How to deal with a small extra cluster in a tabular data?

I am working on a high dimensional tabular dataset with 1600 features and 9440 rows. No matter how I select the features, when I try to project my data into a 2d or 3d graph using dimensionality ...
Tanmay Sharma's user avatar
0 votes
1 answer
512 views

Outlier filtering from time series data

I have time series data that I eventually want to cluster after using dimensionality reduction. I am thinking about how to handle outliers. The data has seasonal/periodic patterns. I have tried IQR ...
Jim A's user avatar
  • 1
0 votes
0 answers
29 views

Do outlier data in target variable cause a problem for GBM's?

I wonder if I should exclude outlier (legit data, not wrong readings) data from my dataset using gradient boosting. Let's say we try to predict water damage for regular houses and 99% of data is in 0-...
morqueatsz's user avatar
0 votes
0 answers
12 views

Using nearest neighbor in RANSAC

I found many resources online talking about nearest neighbor concept in RANSAC. For example, figure 2 of this paper, this article and this repo talk about nearest neighbor in the context of RANSAC. ...
RajS's user avatar
  • 105
1 vote
1 answer
204 views

Bivariate Outlier Detection

I'm trying to find outliers in a bivariate data set. One of the input variables is the amount of time that it takes a user to do a task in our system in seconds (rounded to integer). Most interactions ...
skustes's user avatar
  • 13
0 votes
0 answers
27 views

Binary classification using xgboost

Why when adding new features in my ADS for a binary classification using XGBOOST my score and uplift has decreased ? What is the best way to treate categorical features or other features in order that ...
Warda_IDRIS's user avatar
0 votes
1 answer
207 views

Remove outlier from image array in Python

Could someone please suggest me what would be the best way to remove such huge number of outlier data from the image. The regular clipping between data range in numpy array would simply reduce the ...
hillsonghimire's user avatar
0 votes
1 answer
298 views

How would normalizing be affected by outliers? And how to avoid it?

I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction. I'm trying to find the best way to rank all suppliers ...
Rakuzan's user avatar
0 votes
2 answers
4k views

ML regression for skewed data

I'm trying to build a simple regression model to start with but my Y variable is very skewed to the right. My Y represents the number of views per day for a webpage and all the values are above 0. I ...
Karolina's user avatar
0 votes
1 answer
51 views

Finding outlier for millions of data that can be used cluster-based algorithm

I have millions of data that each have many features. But as long as the value of a feature is not in the acceptable range, that data will be considered an outlier. And I need to find the acceptable ...
Geoffrey's user avatar
1 vote
1 answer
75 views

Outliers: How to handle data values of future

I am learning ML. I have a dataframe with some features and a target column. For simplicity condiser X has one feature, for eg X ...
winter's user avatar
  • 125
0 votes
1 answer
85 views

How to apply a tree-based model with numerical and categorical values to find outliers

I have a dataset which has a column of prices, a column of dates, and various other columns of numerical and categorical values. I would like to find outlier prices based on all the columns in the ...
ImNotSureAboutStats's user avatar
1 vote
1 answer
102 views

Dealing with outliers

I'm doing some data analysis on the UCI "Adult Dataset". I have a numerical feature called "hours-per-week" and another feature called "age". These are the only numerical ...
Beatriz Gonçalves's user avatar
0 votes
1 answer
92 views

Testing RANSAC regression model

I am going to build the model (e.g. multiple linear regression) to predict the appartment cost in my city. First I have to find outliers in training data. For this task RANSAC regression algorithm ...
Irina Svist's user avatar
1 vote
1 answer
75 views

Python Library Trend time series multivariate

Our csv contains 36 columns 1 date time column collected every 30 mins 3 variables (count,latency,Totaltime) x 10 Features(user io, serverio ,concurrency ..etc ) Of different data points from the ...
trent's user avatar
  • 11
0 votes
0 answers
18 views

Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?

I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case? The system I am building assigns classes (-1, 0, +1) to observations ...
HackJob99's user avatar
6 votes
2 answers
2k views

Remove or not to remove outliers

Are there any known academic sources that point towards supporting not removing outliers? Let say if the outlier is a natural occurrence or it has relationship to the value of target variable
Kusisi Karem's user avatar
2 votes
1 answer
2k views

Can GridSearchCV be used for unsupervised learning?

im trying to build an outlier detector to find outliers in test data. That data varies a bit (more test channels, longer/shorter testing). First im applying the train test split because i want to use ...
arooki's user avatar
  • 23
0 votes
1 answer
99 views

what is the difference between 'object detection' and 'outlier detection' in computer vision?

If I'm looking at drone footage, and I'm looking for tennis courts, then I'm doing object detection. If I decide that a tennis court is an 'outlier' as opposed to the rest of the landscape, am I now ...
tumultous_rooster's user avatar
0 votes
0 answers
15 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
Art's user avatar
  • 1
0 votes
2 answers
1k views

How much percentage of outliers are allowed in a data

I am buliding a machine learning model with logistic regression. I am dealing with blood transfusion data set. In which there is a feature,Total_volume, I found that there is more than 5% of outliers ...
Neenu Prasad's user avatar
0 votes
2 answers
94 views

Anomaly prediction/forecasting in timeseries?

What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
M Reyes's user avatar
0 votes
1 answer
230 views

MAE vs MSE for linear regression

Several articles says that MAE is robust to outliers but MSE is not and MSE can hamper the model if errors are too huge. My question is that MSE and MAE both are error matrices,our priority is to just ...
Parth Sharma's user avatar
0 votes
1 answer
102 views

How to deal with imbalanced categorical variables in regression tasks?

I want to predict real estate prices using several Machine Learning algorithms. My dataset contains numerical and categorical predictors. I already eliminated the outliers of numerical variables. Now ...
moby1209's user avatar
0 votes
1 answer
663 views

How do outliers and missing values impact these classifiers?

I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class. I know how to handle finding and ...
Vishnu dut's user avatar
1 vote
1 answer
158 views

How variable alpha changes SGDRegressor behavior for outlier?

I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
Ross_you's user avatar
  • 111
1 vote
2 answers
65 views

Given daily sequence of events with only event ID labels (alphanum strings), what algorithms can be used to detect sequences that are outliers?

For example, the data might be something like this: ...
demoman's user avatar
  • 11
0 votes
1 answer
133 views

Where can I practice multivariate outlier detection?

Can anyone provide me with a dataset, hopefully on Kaggle, where I can practice my skills in outlier analysis? I have been studying this topic for quite a while, but I can't find a case study to apply ...
Mina Ashraf's user avatar
0 votes
1 answer
129 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
TheGoat's user avatar
  • 281
0 votes
3 answers
1k views

How to remove outliers properly?

I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a ...
Erik M's user avatar
  • 83
0 votes
1 answer
1k views

Explanation of spectral residual algorithm for outlier detection

I've been reading the paper https://arxiv.org/pdf/1906.03821.pdf for spectral residual outlier detection, but I don't quite understand it. Specifically, in the implementation there are three variables ...
ptushev's user avatar
  • 11
0 votes
1 answer
70 views

Outlier detection - missing values

I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
Alex's user avatar
  • 1
0 votes
1 answer
547 views

How to gps data anomaly detection in python

I have gps format dataset lat, lon. I want to detection anomaly using python. I tested knn, smv, cof, iforest using pycaret. But i did not. These colors anomlay because the angle change is too much ...
ai-mcv's user avatar
  • 1
3 votes
2 answers
62 views

How to detect whether an entire series is an outlier relative to others?

I have multiple price series of the same asset as follows. Visually, it is obvious that series "A" (the flat line) is an outlier, and series "E" (the line with the zig-zag pattern)...
finstats's user avatar
  • 131
0 votes
1 answer
52 views

Assign a risk score in records in a dataset

I was wondering, if I have a dataset with categorical and numerical data and labels such as 1 or 0 that shows if a row is anomalous or normal respectively. Is it possible to create somehow a model ...
nameguest's user avatar
0 votes
1 answer
108 views

Can I leave natural outliers in a dataset in training?

Can I leave unedited natural outliers in a dataset (outliers that have not appeared just because of mistyping of mistakes in the data)? Or should I also remove them or change them?
Zexxxx's user avatar
  • 33
1 vote
0 answers
120 views

Nonparametric Outlier Detection

Which Nonparametric outlier detection do you suggest to detect outliers (red points) in these plots? I have tested std, IQR, etc., but no good result. It is just one vector including normal and ...
Arkan's user avatar
  • 453
0 votes
1 answer
89 views

Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
spectre's user avatar
  • 2,125
2 votes
1 answer
904 views

Visualizing outliers using T-SNE

I'm trying to visualize outliers in my data using T-SNE and it seems like the outliers appear as three different clusters. The original data has 7 different columns but I chose to plot the outliers on ...
Sarah Grimes's user avatar
0 votes
1 answer
2k views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
letdatado's user avatar
  • 113
0 votes
2 answers
850 views

Removing outliers from a multi-dimensional dataset & Data augmentation

Removing the outliers of a single-dimensional data can be easily done by removing the points that are outside of the IQR range. But how should the process of detecting and removing outliers be done if ...
Centauri_42's user avatar

1
2 3 4 5