1
$\begingroup$

I've been experimenting with Random Forests on Python after trying Naive Bayes which gave me lower accuracy than I expected, 62%. My csv file has around 14,000 records, I use 80% for the training set and 20% for testing set. I tried with different parameters like 100 trees, 500 and 1000, -1 for n_jobs and so on but during all those tests the accuracy never changed too much, it was always around 74% or 75%, almost 76% at times. I checked that the online tutorial which teaches how to implement got the accuracy of 73%, as my tests got higher than that, I wasn't worried, thought it's normal for this kind of algorithm.

However, my surprise came during my latest test, the only difference is I saved the model and vectorizer to file so I don't have to train the model each time. The new script loaded those files, applied the loaded (trained) model to the testing set and wow, I got the highest accuracy I've seen so far, 93.989%, almost 94%.

Anyone got a similar experience? Is this big increase related to having saved the model to a file and loaded it? The rest of the code is all the same. Or was I just too lucky and if I try more times I will go back to the usual around 75% accuracy?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Just saving and loading a file won't increase the model's test accuracy. Share your code please. $\endgroup$
    – Yash Jakhotiya
    Commented Jul 7, 2019 at 19:31
  • 1
    $\begingroup$ Your 94% performance looks like there was a mix-up between the training set and the test set. $\endgroup$
    – Erwan
    Commented Jul 7, 2019 at 19:54
  • $\begingroup$ @Ewan after giving it some thought, maybe that's the issue. When I trained the model, I used train_test_split() to split the original dataset into a random 20% for tests and 80% for training. Then saved the model to file. In my new script, my goal was to just load the model from file and apply it to any dataset. In the 2nd script, when I use train_test_split() again, I obtain a different 20% set for tests, it can overlap with many of the records I used before to train, since it's random. So for the model, it's like cheating, I already gave him the answers. $\endgroup$
    – AndroidMarshmallow
    Commented Jul 7, 2019 at 20:35
  • $\begingroup$ I can't use the model loaded from file on any dataset? In the new script, I don't have the same testing dataset from train_test_split() and don't know which one was used during training, I have another one with another 20%. Won't any dataset work well? How to get one that doesn't overlap with the 80% I used for training? $\endgroup$
    – AndroidMarshmallow
    Commented Jul 7, 2019 at 20:37

2 Answers 2

Reset to default
1
$\begingroup$

You can be sure that saving the model to disk won't make the model have higher accuracy when loaded again. Saving the models to disk would only cause worse performance if parameters are truncated, for example.

To establish a fair and rigorous comparison method, you have to keep track of your training/test splitting result. In that sense, you can:

  • Make your code reproducible, using random seeds conveniently
  • Before saving the model, you can save your splitting results by either saving the data or returning the indices for the training set and the test set. For that purpose you have to store the data-set in a list (for example, a list with the file names…)
Improve this answer
$\endgroup$
2
  • $\begingroup$ I figured the higher accuracy must come from the fact I was using the model loaded from file on all the data including the 80% used for training, so the model already knows the answers for those 80%, it's like cheating. Should I only use a model loaded from file on new data? The accuracy method only makes sense when training? $\endgroup$
    – AndroidMarshmallow
    Commented Jul 8, 2019 at 11:05
  • $\begingroup$ The accuracy is a metric, you can use it on whatever set you want, but you have to know the meaning of the accuracy in each set... you can check my previous posts datascience.stackexchange.com/a/31664/51714 and datascience.stackexchange.com/a/31626/51714 $\endgroup$
    – ignatius
    Commented Jul 8, 2019 at 11:11
1
$\begingroup$

One way to get an unbiased accuracy estimate is to combine your entire dataset, run random forest (incorporating bagging, as most algorithms do anyway), and report the random forest out-of-bag (OOB) accuracy (see oob_score_ if using sklearn) as an estimate of the true accuracy (as suggested by Breiman see sec 3.1) . You can do this so that your training set doesn't overlap with your testing set. This OOB can be used as an estimate of accuracy.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Richard, my trained forest saved to file had oob_score set to False, so I retrained from scratch with oob_score as True, then saved to file again, loaded from another script, checked oob_score and it showed 75%, which is the same accuracy value I got when training from the accuracy() method. I've read a bit about oob_score (out of bag score) but your comment makes sense, as the value coincided with the training accuracy. $\endgroup$
    – AndroidMarshmallow
    Commented Jul 9, 2019 at 11:14
  • $\begingroup$ I just gave to my random forest this text: "AI is amazing and it's going to take over the world.", the forest predicted correctly: positive. When I changed the sentence to "AI is bad", the forest predicted: negative. Interesting lol very smart forest. $\endgroup$
    – AndroidMarshmallow
    Commented Jul 9, 2019 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.