# Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension.

For example:

: [10, 12, 13, 15, 16] indicates that the sensor  will have some operating duration like [10, 12, 13, 15, 16].

I want to see the range of operating duration for each sensor, by measuring the mean and standard deviation for each sensor. But my problem is in my dataset, each sensor has many noises. For example:

: [1, 1, 1, 1, 1, 2, 2, 2, 10, 12, 13, 15, 16, 200, 400, 500].


You could see that sensor  has many noises like 1, 2, 200, 400 and 500. In my dataset, there are many cases like this.

Without removing the noise, the standard deviation is always smaller than the mean. This makes my duration analysis not meaningful.

So my question is: I want to ask if there is any method for removing the noises like that in my dataset.

Thank you.

## 1 Answer

There are multiple algorithms for this. On one of the kaggle competitions I saw applying several algorithms and then throwing the data away by the number of votes from all algorithms. The algorithms that I remember:

1. Remove all points above or below 2 standard deviations. X>mean+2*std dev, X < mean-2*std dev. 2 is actually a parameter you need to optimize.
2. Remove the 5% of the highest and lowest values. X>95% quantile, X<5% quantile. 5% is a parameter you need to optimize.
3. X - median > 2*1.4826*MAD (median absolute deviation. 1.4826*MAD is an estimator for standard deviation under some assumptions). And the same for lower bound. https://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/

Repeat several times until no points are thrown away.

Example of this method with voting from some other kaggle winners: https://nycdatascience.com/blog/student-works/kaggle-predict-consumer-credit-default/

• Thank you for your suggestion. I think it is quite suitable for my problem now. – Truong Nguyen Dec 3 '18 at 7:02