2
$\begingroup$

I have one image that contains for each pixel 4 different values. I have used RF in order to see if I can predict the 4th value based on the other 3 values of each pixel. for that I have used python and scikit learn. first I have fit the model, and after validate it I used it to predict this image. I was very happy and scared to see that I got very high accuracy for my model : 99.95%! but then when I saw the resulted image it absolutly wasn't 99.95% of accuracy:

original image:

enter image description here

result image:

enter image description here

(I have makrd the biggest and most visible difference).

My question is- why would I get this high accuracy when the visualization shows very well that there is much less accuracy? I understand it might come from overfitting but then how this different is not detected?

edit: Mean Absolute Error: 0.048246606512422616 Mean Squared Error: 0.00670919112477127 Root Mean Squared Error: 0.0819096522076078 Accuracy: 99.95175339348758

$\endgroup$
3
  • $\begingroup$ MAE/MSE/RMSE are metrics for regression, while Accuracy is for classification. What are you measuring with Accuracy? $\endgroup$ – Carlos Mougan Jun 21 '20 at 13:52
  • $\begingroup$ I would like to get some measurment of how good was the classification $\endgroup$ – Reut Jun 21 '20 at 14:14
  • $\begingroup$ But you are running a regression model, the post is called random forest regressor. Accuracy is not a great metric to evaluate this. What is your train target like? $\endgroup$ – Carlos Mougan Jun 21 '20 at 15:52
1
$\begingroup$

Where are you evaluating the performance of your algorithm?

Are you making a train test split and evaluating in the test split? It might be that you overfitted your train and you are just measuring the accuracy there.

If you have made correctly the train/test split and the evaluation it could be that the images that you are predicting do not have the same properties/configuration/topology than the with you are trainning

$\endgroup$
4
  • $\begingroup$ I do use the test split and then I fit the model and predict it. I do all those calculations in pandas table, so each pixel is a row, and there not suppose to be influence of topology or coordinates $\endgroup$ – Reut Jun 21 '20 at 13:32
  • $\begingroup$ whats your accuracy in test? can you provide this output in train and test? scikit-learn.org/stable/modules/generated/… $\endgroup$ – Carlos Mougan Jun 21 '20 at 13:39
  • $\begingroup$ By eye I would guess taht the accuracy is more or less 90% in test $\endgroup$ – Carlos Mougan Jun 21 '20 at 13:41
  • $\begingroup$ @I have added this data in the original post right now :) $\endgroup$ – Reut Jun 21 '20 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.