I am trying to validate the accuracy of my KNN algorithm for the movie rating prediction.

I have $2$ vectors: $Y$ - with the real ratings, $Y'$ - with predicted ones.

When I calculate Standard Error of the Estimate (is it the one I need to calculate?) using following formula:

$$\sigma_{est} = \sqrt{\frac{\sum (Y-Y')^2}{N}}$$

I'm getting result of $\sim 1.03$. But I thought that it can't be $> 1$. If it is not, then what does this number say to me?

results = load('first_try.mat');
Y = results(:,1);
Y_predicted = results(:,2);

o = sqrt(sum((Y-Y_predicted).^2)/rows(Y))
  • 2
    $\begingroup$ Why did you think it can't be bigger than 1? $\endgroup$ Oct 2 '18 at 6:54
  • $\begingroup$ I thought that 1 means that 100% match $\endgroup$ Oct 2 '18 at 6:54
  • 2
    $\begingroup$ That's not what a standard error (SE) is, the SE is the standard deviation (SD) of your estimate, which goes from 0 to infinity. $\endgroup$ Oct 2 '18 at 6:55
  • 1
    $\begingroup$ Depends what you mean by accuracy, the mean is usually used as your accuracy measure, the SE is used to show how variable this measure is (for ex. building a 95 % CI). Whether 1.03 is big in your case depends entirely on your data. $\endgroup$ Oct 2 '18 at 7:10
  • 2
    $\begingroup$ "I thought that 1 means that 100% match" if we had a perfect match (i.e. $Y=Y'$), then wouldn't the numerator of your expression equal $0$ and hence give you $\sigma_{est}=0$? $\endgroup$ Oct 2 '18 at 8:30

K-NN is a measure of distance, thus the result of your equation will depend on the scale of your data. If the ratings are in a scale from 0 to 100. Then if you always predict very poorly you are evidently going to have values much larger than 1.

For example, for a very bad predictor

import numpy as np

Y = [100, 90, 100, 90]
Y_p = [10, 10, 10, 10]

np.sqrt(np.sum(np.subtract(Y, Y_p)**2)/len(Y))



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.