3
$\begingroup$

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the exact same values as per the training data. However, my test tuples can have values that exceeds the maximum value of the data in the training set. For example, in the training set I have following values for a given feature.

Maximum : 20457
Minimum: 3

In the testing tuple, I sometimes get values like 35002. This is present in most of the features.

The problem would have been solved if I knew the maximum and minimum values for all feature, but it no possible. The maximum value can go up to any value. How do I do data normalization in a scenario like this? Can someone please help me with this?

$\endgroup$
2
$\begingroup$

Judging from your question you are probably using this formulae for normalisation:

(x - x_mean)/(x_max - x_min). This is just an approximation of the real normalization formulae. The real one would be:

enter image description here

where mu is the mean and sigma is standard deviation. If your data trends is same throughout, then you can expect the mean and standard deviation to be approximately same, and thus give you a more uniform representation.

Check this Wikipedia article, Feature scaling which says about the schemes used in different ML techniques. Hope this helps!

$\endgroup$
  • $\begingroup$ Yes you are correct, I used min max normalization. Thanks a lot, this is what I was looking for! $\endgroup$ – harsh Jul 1 '18 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.