1
$\begingroup$

I am trying to validate the accuracy of my KNN algorithm for the movie rating prediction.

I have $2$ vectors: $Y$ - with the real ratings, $Y'$ - with predicted ones.

When I calculate Standard Error of the Estimate (is it the one I need to calculate?) using following formula:

$$\sigma_{est} = \sqrt{\frac{\sum (Y-Y')^2}{N}}$$

I'm getting result of $\sim 1.03$. But I thought that it can't be $> 1$. If it is not, then what does this number say to me?

results = load('first_try.mat');
Y = results(:,1);
Y_predicted = results(:,2);

o = sqrt(sum((Y-Y_predicted).^2)/rows(Y))
Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Why did you think it can't be bigger than 1? $\endgroup$
    – user2974951
    Oct 2, 2018 at 6:54
  • $\begingroup$ I thought that 1 means that 100% match $\endgroup$
    – Dmitrij Kultasev
    Oct 2, 2018 at 6:54
  • 2
    $\begingroup$ That's not what a standard error (SE) is, the SE is the standard deviation (SD) of your estimate, which goes from 0 to infinity. $\endgroup$
    – user2974951
    Oct 2, 2018 at 6:55
  • 1
    $\begingroup$ Depends what you mean by accuracy, the mean is usually used as your accuracy measure, the SE is used to show how variable this measure is (for ex. building a 95 % CI). Whether 1.03 is big in your case depends entirely on your data. $\endgroup$
    – user2974951
    Oct 2, 2018 at 7:10
  • 2
    $\begingroup$ "I thought that 1 means that 100% match" if we had a perfect match (i.e. $Y=Y'$), then wouldn't the numerator of your expression equal $0$ and hence give you $\sigma_{est}=0$? $\endgroup$
    – ignoring_gravity
    Oct 2, 2018 at 8:30

1 Answer 1

Reset to default
1
$\begingroup$

K-NN is a measure of distance, thus the result of your equation will depend on the scale of your data. If the ratings are in a scale from 0 to 100. Then if you always predict very poorly you are evidently going to have values much larger than 1.

For example, for a very bad predictor

import numpy as np

Y = [100, 90, 100, 90]
Y_p = [10, 10, 10, 10]

np.sqrt(np.sum(np.subtract(Y, Y_p)**2)/len(Y))

85.14693182963201

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.