0
$\begingroup$

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension.

For example:

[1]: [10, 12, 13, 15, 16] indicates that the sensor [1] will have some operating duration like [10, 12, 13, 15, 16].

I want to see the range of operating duration for each sensor, by measuring the mean and standard deviation for each sensor. But my problem is in my dataset, each sensor has many noises. For example:

[1]: [1, 1, 1, 1, 1, 2, 2, 2, 10, 12, 13, 15, 16, 200, 400, 500].

You could see that sensor [1] has many noises like 1, 2, 200, 400 and 500. In my dataset, there are many cases like this.

Without removing the noise, the standard deviation is always smaller than the mean. This makes my duration analysis not meaningful.

So my question is: I want to ask if there is any method for removing the noises like that in my dataset.

Thank you.

$\endgroup$
0
$\begingroup$

There are multiple algorithms for this. On one of the kaggle competitions I saw applying several algorithms and then throwing the data away by the number of votes from all algorithms. The algorithms that I remember:

  1. Remove all points above or below 2 standard deviations. X>mean+2*std dev, X < mean-2*std dev. 2 is actually a parameter you need to optimize.
  2. Remove the 5% of the highest and lowest values. X>95% quantile, X<5% quantile. 5% is a parameter you need to optimize.
  3. X - median > 2*1.4826*MAD (median absolute deviation. 1.4826*MAD is an estimator for standard deviation under some assumptions). And the same for lower bound. https://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/

Repeat several times until no points are thrown away.

Example of this method with voting from some other kaggle winners: https://nycdatascience.com/blog/student-works/kaggle-predict-consumer-credit-default/

$\endgroup$
  • $\begingroup$ Thank you for your suggestion. I think it is quite suitable for my problem now. $\endgroup$ – Truong Nguyen Dec 3 '18 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.