I've been experimenting with Random Forests on Python after trying Naive Bayes which gave me lower accuracy than I expected, 62%. My csv file has around 14,000 records, I use 80% for the training set and 20% for testing set. I tried with different parameters like 100 trees, 500 and 1000, -1 for n_jobs and so on but during all those tests the accuracy never changed too much, it was always around 74% or 75%, almost 76% at times. I checked that the online tutorial which teaches how to implement got the accuracy of 73%, as my tests got higher than that, I wasn't worried, thought it's normal for this kind of algorithm.
However, my surprise came during my latest test, the only difference is I saved the model and vectorizer to file so I don't have to train the model each time. The new script loaded those files, applied the loaded (trained) model to the testing set and wow, I got the highest accuracy I've seen so far, 93.989%, almost 94%.
Anyone got a similar experience? Is this big increase related to having saved the model to a file and loaded it? The rest of the code is all the same. Or was I just too lucky and if I try more times I will go back to the usual around 75% accuracy?