Newest 'outlier' Questions - Page 3 - Data Science Stack Exchange

0

votes

2answers

121 views

How to make multiple regression perform better for outliers? (without reducing effect of them)

I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values. So I ...

asked Oct 5 '18 at 12:39

DmytroSytro

4472 silver badges11 bronze badges

0

votes

2answers

306 views

Replace Values in Vector on Specific Place in R

I want to make $5^{th}$,$10^{th}$,$15^{th}$,$20^{th}$ and $25^{th}$ values of vector an outlier in all xs by using x1 [5]+OT1,...

r statistics outlier

asked Oct 5 '18 at 9:11

Angel

366 bronze badges

2

votes

1answer

225 views

Probabilistic Outlier Detection (edited + clarified)

Measured data $D \in \mathbb{R}^3$, every $d^i \in D$ is $d^i_{(x)}$, where the $x=[x_1, x_2]$. Simply said, the measured data are function of $x$. It is known, the dependency is linear, such as: $$...

predictive-modeling outlier gaussian

asked Oct 4 '18 at 5:30

Martin G

1213 bronze badges

1

vote

0answers

137 views

Can we use doc2vec to detect outlier documents?

I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task. Or are there any recently evolved, promising algorithms ...

data-mining nlp word2vec outlier gensim

asked Sep 25 '18 at 5:34

J Cena

1911 silver badge7 bronze badges

1

vote

1answer

27 views

Should we identify outliers from population prior to taking sample?

I am revising undergrad statistics course via this course, where i am learning techniques to pull out sample from population. While ensuring that sample is a decent representative of the population, ...

machine-learning statistics outlier data-science-model

asked Aug 6 '18 at 7:02

Gaurav Gupta

1972 silver badges7 bronze badges

1

vote

0answers

42 views

Taking advatage of the frequency of an outlier

I'm doing outlier detection (Conditional Outliers) on a multivariate time series. The outliers appear every 2 weeks $\pm$ 4 days. How can I incorporate this prior in my models, to reduce the number ...

predictive-modeling time-series outlier

asked Jul 11 '18 at 17:43

Databoi

114 bronze badges

2

votes

3answers

663 views

Why replacing null values with outliers?

I have been watching a tutorial on stock price prediction with multivariate linear regression and the tutor replaces missing value data, NaN, with the outlier -99999. Why and how do replacements like ...

machine-learning regression linear-regression outlier machine-learning-model

asked Jul 1 '18 at 15:04

maufcost

1211 bronze badge

1

vote

1answer

16k views

Remove Local Outliers from Dataframe using pandas

Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe....

pandas outlier dataframe

asked Jun 25 '18 at 18:05

natg

151 gold badge2 silver badges5 bronze badges

1

vote

1answer

757 views

LOF gives same number of outliers irrespective of parameters

I am running lof algorithm for around 100k 2d points. Each time, I run the lof algorithm with different n_neighbours parameter, I get the same number of points as outliers. It's always 10% of the ...

python scikit-learn clustering outlier

asked Jun 11 '18 at 12:16

ayush gupta

1133 bronze badges

0

votes

2answers

68 views

Unsupervised Clustering of 6-d vectors

I have 17k vectors, each having 6 points, I want to cluster the vectors based on nature of points, eg. linearly increasing into one cluster, convex in another cluster, concave in another, decreasing ...

python clustering regression outlier

asked Jun 5 '18 at 14:18

ayush gupta

1133 bronze badges

0

votes

1answer

4k views

Implement Sliding Window Algorithm to Detect Spikes

How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.

python outlier dataframe

asked May 31 '18 at 16:14

natg

151 gold badge2 silver badges5 bronze badges

3

votes

2answers

1k views

Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the ...

machine-learning scikit-learn anomaly-detection outlier

asked May 31 '18 at 9:02

David Masip

5,2211 gold badge15 silver badges48 bronze badges

1

vote

2answers

1k views

Using local outlier factor score to detect outliers at run time

I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my ...

clustering anomaly-detection outlier

asked May 18 '18 at 15:21

Arpit Sisodia

3751 silver badge9 bronze badges

4

votes

3answers

6k views

Handling outliers and Null values in Decision tree

Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example? (What if I have a features ranging from 0 to 9 but there is an ...

classification regression decision-trees outlier

asked May 9 '18 at 12:18

deepguy

1,3115 gold badges14 silver badges34 bronze badges

-1

votes

1answer

145 views

Understanding a Box plot

I have a dataset and I have tried plotting it in a box plot after loading the csv in usings pandas. The plot itself is given below. I need to understand why the x1 ...

visualization outlier

asked May 8 '18 at 4:50

nainometer

1214 bronze badges

1

vote

1answer

125 views

Principal Component Analysis and abnormal data

I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow ...

machine-learning dimensionality-reduction pca anomaly-detection outlier

asked Apr 25 '18 at 4:47

Arkan

4332 silver badges12 bronze badges

4

votes

1answer

5k views

Multivariate outlier detection with isolation forest..How to detect most effective features?

I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/...

python scikit-learn anomaly-detection outlier

asked Apr 23 '18 at 14:33

mlee_jordan

1311 silver badge6 bronze badges

2

votes

2answers

287 views

General methods outlier detection

What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of ...

machine-learning statistics anomaly-detection outlier

asked Apr 19 '18 at 10:52

David Masip

5,2211 gold badge15 silver badges48 bronze badges

0

votes

1answer

37 views

outlier detection in time serie without using windows [closed]

I would like to know if it's possible to detect outliers in a time-serie with an outlier score computed given the whole dataset and not given windows

time-series outlier

asked Mar 27 '18 at 13:40

youngam

517 bronze badges

1

vote

0answers

353 views

Isolation forest results every value -1

I am trying out isolation forest to detect outliers in a specific target column of my dataset. The dataset contains 188 rows of data with 178 rows with the same value for that target column and the ...

python scikit-learn pandas outlier

asked Mar 19 '18 at 2:53

Suraj Motaparthy

1514 bronze badges

4

votes

2answers

4k views

When to remove outlier in preparing features for machine learning algorithm

I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, ...

data-cleaning outlier

asked Mar 6 '18 at 3:02

KevinKim

4551 gold badge5 silver badges12 bronze badges

1

vote

1answer

259 views

An outlier detection for this data

I have a vector and want to detect outliers in it. I need an outlier detection method (a non-parametric method) which can detect red points as outliers. Edit: I have a lot of vectors like this. The ...

machine-learning data-mining anomaly-detection outlier

asked Mar 3 '18 at 8:44

Arkan

4332 silver badges12 bronze badges

5

votes

3answers

2k views

To detect unauthorized access using outlier detection

I am working on project where my task is to find unauthorized access using any machine learning technique. Let me clear my problem definition. UserA access website using chrome browser from windows ...

machine-learning python unsupervised-learning anomaly-detection outlier

asked Feb 5 '18 at 13:57

Nilesh Shaikh

495 bronze badges

0

votes

1answer

55 views

Differentiating between normal and attack clusters

In the below figure, we plotted some data of sensors in normal condition and under attack(outlier): 1. Green points are normal samples in the training dataset. 2. Cyan and red points are normal ...

machine-learning classification clustering visualization outlier

asked Jan 20 '18 at 16:17

Arkan

4332 silver badges12 bronze badges

1

vote

1answer

754 views

Isolation Forest height limit absent in SkLearn implementation

In the original publication of the Isolation Forest algorithm, the authors mention a height limit parameter to control the granularity of the algorithm. I did not find that explicit parameter on the ...

scikit-learn decision-trees outlier

asked Jan 15 '18 at 15:42

jimijazz

1565 bronze badges

1

vote

4answers

461 views

How to detect anomalous points

As is clear from the figure, the blue points, which don't follow the trend, are anomalous points. I'm wondering about the best non-parametric method to detect those points. I have tested some ...

statistics visualization anomaly-detection outlier anomaly

asked Dec 14 '17 at 2:30

Arkan

4332 silver badges12 bronze badges

1

vote

0answers

366 views

Sensitivity analysis in outlier explanation

I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are ...

python data-mining bigdata outlier

asked Nov 30 '17 at 11:23

Shashank Srivastava

214 bronze badges

10

votes

3answers

6k views

What is the difference between outlier detection and anomaly detection?

I would like to know the difference in terms of applications (e.g. which one is credit card fraud detection?) and in terms of used techniques. Example papers which define the task would be welcome.

algorithms anomaly-detection outlier terminology definitions

asked Nov 15 '17 at 11:17

Martin Thoma

15.9k23 gold badges82 silver badges157 bronze badges

1

vote

2answers

2k views

How to decide for the contamination value (proportion of the outliers) in my dataset?

I should decide on the contamination value while using the Isolation Forests algorithm (I am using the sklearn implementation). Otherwise, sklearn's default is 0.1. I am worried if I decide for this ...

data-cleaning outlier

asked Nov 14 '17 at 15:11

mari

1211 silver badge8 bronze badges

2

votes

0answers

86 views

Outlier detection: Should the metric used in kNN take into account variance explained by each coordinate?

After applying PCA and working with the reduced dataset, I want to delete the outliers. To do this my idea is to compute the kNN-graph and delete those vertices (points) that have an inner degree of 0....

outlier distance

asked Oct 18 '17 at 15:04

h3h325

2431 silver badge6 bronze badges

3

votes

2answers

891 views

How to increase the weight when it comes to outlier detection

Let's say we have feature A, B, C, D, E to represent one observation in an outlier detection model. We are using scikit-learn outlier detection in our case. AFAIK, if we normalize all the features, ...

scikit-learn outlier

asked Sep 12 '17 at 7:52

Shengjie

2212 silver badges6 bronze badges

6

votes

2answers

913 views

Anomaly detection in time series

The use case : Everyday, we have metrics that are established daily to check that our systems are doing fine. From time to times, bugs occur in the workflow building these metrics, and I have to ...

r time-series outlier

asked Aug 30 '17 at 10:03

Den

611 bronze badge

1

vote

1answer

2k views

Can the Generative Adversarial Network useful for Outlier detection and Outlier explanation in a high dimentional numerical data?

I have been building a model to find explanation of Outliers in a high dimensional numerical data, generated from many sensors. The data contains more than 350 different fields and each field has ...

neural-network data-mining deep-learning outlier gan

asked Aug 15 '17 at 12:18

Shashank Srivastava

214 bronze badges

2

votes

1answer

1k views

Outlier detection on categorical network log data

I am working with a completely categorical network log data that consists of source ip address, destination ip address, source port, destination port, protocol. Data Preprocessing performed : ...

r clustering categorical-data unsupervised-learning outlier

asked Jul 20 '17 at 19:19

iprof0214

1511 gold badge1 silver badge6 bronze badges

1

vote

2answers

127 views

Notion of cluster centers and cluster comparison in Density Based Algorithms

I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main ...

python clustering outlier clusters

asked Jul 17 '17 at 8:52

MikEKOU

475 bronze badges

1

vote

1answer

2k views

HDBSCAN Outlier Detection and labeling

I am using the HDBSCAN algorithm so as to perform unsupervised clustering and detect outliers. Based on the documentation there are two outputs from the clustering process that can give insight on ...

python clustering outlier

asked Jul 9 '17 at 13:45

MikEKOU

475 bronze badges

1

vote

2answers

2k views

Classification affected by a lot of outliers in features? How do you deal with outliers?

I am working on a classification problem and I found my data having a lot of outliers which has resulted in reduction in my recognition rate. I have tried rescaling, normalization techniques like ...

multiclass-classification outlier normalization

asked May 7 '17 at 22:49

sam pi

112 bronze badges

-1

votes

1answer

132 views

Considering outliers in demand predictions

I have times series data with demand observations during months. I was wondering if, when computing demand predictions, I need to consider the outliers of the observations or not. What is your opinion ...

predictive-modeling prediction outlier

asked May 4 '17 at 14:11

Bernheart

1011 bronze badge

3

votes

3answers

7k views

Difference between Global Outlier and Contextual Outlier?

I am studying "Data Mining: Concepts and Techniques" by Han, Kamber & Pei. In Chapter 12 "Outlier Detection", they have stated that there are 3 types of outliers: Global Outlier - deviates ...

classification data-mining clustering outlier

asked May 1 '17 at 15:04

user31795

1

vote

3answers

595 views

Detecting outlier with combining two vectors

I want to combine the following vectors in a way that just the red point (number 7) becomes inconsistent with other points( become an outlier and become distant from other points) and other points ...

data-mining feature-extraction feature-engineering anomaly-detection outlier

asked Apr 25 '17 at 22:43

Arkan

4332 silver badges12 bronze badges

7

votes

3answers

1k views

Which outlier detection can detect these outliers?

I have a vector and want to detect outliers in it. The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal. ...

unsupervised-learning anomaly-detection outlier distribution

asked Apr 19 '17 at 10:17

Arkan

4332 silver badges12 bronze badges

3

votes

2answers

4k views

Outlier detection by unsupervised algorithm: Fraud Detection

I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset. I have created around 5 features (All continuous data, with ...

python clustering scikit-learn svm outlier

asked Mar 5 '17 at 12:17

Neil

1672 silver badges4 bronze badges

3

votes

2answers

802 views

Which Outlier Detection Method? Why?

For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a ...

statistics anomaly-detection outlier

asked Feb 20 '17 at 21:05

Arkan

4332 silver badges12 bronze badges

0

votes

1answer

54 views

Outliers Approach

Having a schema which the majority of the values are IDs. Like this example (this isn't my real data): ...

preprocessing outlier

asked Dec 9 '16 at 13:53

Pedro_Rodgers

53 bronze badges

0

votes

1answer

84 views

How can we detect the existence of outliers using mean and median?

How can we detect the existence of outliers using mean and median? Is it really possible to detect the existence of outliers in a set of data from their feature-wise mean and median? Suppose, I have ...

feature-selection outlier

asked Nov 26 '16 at 5:50

user9232

2

votes

2answers

765 views

Outlier detection and removal using scatter-plots and histograms

Suppose, we use the following code to generate scatter plots, ...

machine-learning matlab naive-bayes-classifier outlier

asked Nov 15 '16 at 21:07

user9232

2

votes

1answer

393 views

Remove Outliers - Market Basket Analysis

I'm having some thoughts on whether I should remove the outliers. I'm trying to find the tags that are commonly used together. Imagine that I have the following dataset. The first column is the Tag_ID ...

data-cleaning association-rules outlier

asked Nov 14 '16 at 19:16

Pedro Alves

3472 gold badges3 silver badges11 bronze badges

2

votes

2answers

3k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...

data-mining dataset data outlier

asked Sep 23 '16 at 23:27

siddshah

211 silver badge3 bronze badges

2

votes

1answer

101 views

Some algorithms and approach for identification of specific patterns?

I am working on a hobby project where I have a data related to financial trades(such as stock trades). Now I want to analyse this data and extract possible ...

machine-learning outlier

asked Aug 3 '16 at 1:02

user2916886

1213 bronze badges

4

votes

2answers

81 views

Represent outlier days

I have hourly power consumption data for 30 days. On representing, each day data using a separate line, I get a plot as I want to highlight the days with abnormally high consumption (in other words, ...

time-series visualization outlier

asked Jul 16 '16 at 9:44

Haroon Rashid

2901 silver badge9 bronze badges

Questions tagged [outlier]

Related Tags