working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of input is essential for my dataset.

problem with normalization is that when noise value is very high compare to entire dataset. then prediction follows noise. for example if I add noise (delta=300) to 80% of the data and perform normalization on the dataset which mean value is 250 and standard deviation is 79. here noisy data points(80% of the total dataset) are greater than 300. after normalization, I feed this dataset to the model, prediction follows the noise and giving wrong output. this happens because of feature scaling. when I add noise to most of data points, model consider this points as normal data points and rest of as anomalous data points.

In inverse scaling process, I can not use min-max values of my input to perform inverse scaling on prediction otherwise its follow noise in dataset.

so what is right way to perform feature scaling in denoise kind of problem?

  • $\begingroup$ It would help if you give an exact example of noise addition and normalization for some 1D dataset like (1, 1, 2, 3, 3, 4, 5, 6, 6, 7) $\endgroup$ – Esmailian Mar 12 '19 at 9:44
  • $\begingroup$ I have a multivariate(16 feature) dataset. each feature represent sensor data(temperature, pressure, etc ) of industrial machinery. Training dataset represent normal condition of machinery and goal is to find abnormal behaviour of sensor data. I added some noise(constant) value to some data points. for example I add delta=300 to first 1000 points in one of 16 features. when I feed this perturbed input to model. prediction follows noise instead of giving normal output. $\endgroup$ – Milan_Harkhani Mar 12 '19 at 11:36
  • $\begingroup$ this odd behavior makes sense. Suppose 1000 points are around 0 with standard deviation 1. So if you add 300 to 800 of them, now we have 200 points around 0 and 800 points around 300. This way, these 800 points are the NEW normal, those 200 points become the new outliers! $\endgroup$ – Esmailian Mar 12 '19 at 11:41
  • $\begingroup$ exactly that's what I found. I also have test dataset, sensor data when machine was in faulty condition. its a 1400 points in size. one of sensor output is under 20 for first 800 points which is consider as normal behaviour and then after goes up to 50 which is deviation from normal. when I applied min-max scaling and inverse transform on predicted value with same min-max value. it follows noise and output goes up to above 50 for anomalous data points. so what is right way to perform normalization and denormalization. $\endgroup$ – Milan_Harkhani Mar 12 '19 at 11:59
  • $\begingroup$ The problem is with the amount of noise added, no normalization can make this right since the majority of data (80%) will always stick together under any normalization and always will be the norm. $\endgroup$ – Esmailian Mar 12 '19 at 12:07

Min-Max-Scaling will not perform well for your problem, as you already said. For noisy data, scaling according to the quantile range should perform better.

After scaling, you could also try to clip your data to $[-1,1]$, as it it is often done in adversarial learning methods.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.