2
$\begingroup$

I have one image that contains for each pixel 4 different values. I have used RF in order to see if I can predict the 4th value based on the other 3 values of each pixel. for that I have used python and scikit learn. first I have fit the model, and after validate it I used it to predict this image. I was very happy and scared to see that I got very high accuracy for my model : 99.95%! but then when I saw the resulted image it absolutly wasn't 99.95% of accuracy:

original image:

enter image description here

result image:

enter image description here

(I have makrd the biggest and most visible difference).

My question is- why would I get this high accuracy when the visualization shows very well that there is much less accuracy? I understand it might come from overfitting but then how this different is not detected?

edit: Mean Absolute Error: 0.048246606512422616 Mean Squared Error: 0.00670919112477127 Root Mean Squared Error: 0.0819096522076078 Accuracy: 99.95175339348758

Improve this question
$\endgroup$
3
  • $\begingroup$ MAE/MSE/RMSE are metrics for regression, while Accuracy is for classification. What are you measuring with Accuracy? $\endgroup$
    – Carlos Mougan
    Jun 21, 2020 at 13:52
  • $\begingroup$ I would like to get some measurment of how good was the classification $\endgroup$
    – Reut
    Jun 21, 2020 at 14:14
  • $\begingroup$ But you are running a regression model, the post is called random forest regressor. Accuracy is not a great metric to evaluate this. What is your train target like? $\endgroup$
    – Carlos Mougan
    Jun 21, 2020 at 15:52

1 Answer 1

Reset to default
1
$\begingroup$

Where are you evaluating the performance of your algorithm?

Are you making a train test split and evaluating in the test split? It might be that you overfitted your train and you are just measuring the accuracy there.

If you have made correctly the train/test split and the evaluation it could be that the images that you are predicting do not have the same properties/configuration/topology than the with you are trainning

Improve this answer
$\endgroup$
4
  • $\begingroup$ I do use the test split and then I fit the model and predict it. I do all those calculations in pandas table, so each pixel is a row, and there not suppose to be influence of topology or coordinates $\endgroup$
    – Reut
    Jun 21, 2020 at 13:32
  • $\begingroup$ whats your accuracy in test? can you provide this output in train and test? scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Carlos Mougan
    Jun 21, 2020 at 13:39
  • $\begingroup$ By eye I would guess taht the accuracy is more or less 90% in test $\endgroup$
    – Carlos Mougan
    Jun 21, 2020 at 13:41
  • $\begingroup$ @I have added this data in the original post right now :) $\endgroup$
    – Reut
    Jun 21, 2020 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.