I ran a set of time series data in a neural net model and random forest model. For neural net I normalised the data split it into test 20%, validation and training 80%. Keras library neural net was used here.The accuracy is 85%(I get different accuracy every time). For random forest I didn’t normalise the data and the test train split is 20% and 80%.I used scikit learn here. I am getting 93% accuracy .Here time series data is used as well. Am I not supposed to get similar accuracy as neural net? What am I doing wrong? (I was reading some papers there they had similar kind of accuracy. That’s why I am asking. Also can random forest overfit? Is mine one overfitting?!
2
-
$\begingroup$ It is possible that your net isn't properly tuned but without more info we can't help you. $\endgroup$– Lucas MorinDec 2, 2020 at 7:23
-
$\begingroup$ Can you please tell me how should I check if my neural net is properly tuned? What are the tuning parameters? $\endgroup$– ashDec 2, 2020 at 9:10
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Different models will give different accuracies. Same model too could give you different accuracies. You can try setting the seeds to predefined values to bring in more consistency in the same model results
-
$\begingroup$ Thank you very much. I didn’t set seed as I thought it is a time series data so it always have the same train and test data. Am I missing anything there? $\endgroup$– ashDec 3, 2020 at 7:59
-
$\begingroup$ This is a good place to start: medium.com/@ODSC/… $\endgroup$– AllohvkDec 3, 2020 at 8:57
-
$\begingroup$
$\endgroup$
8
Think of your model as a function that maps input features to the response variable. Random forest classifier/regressor is always a piece-wise constant function. Neural nets are continuous functions (they are the successive implementation of linear maps and continuous activation functions). They are always different, so your work is accurate here.
For consistency issues, try setting up the seed as mentioned above. Also, try adding cross-validation to your model to avoid too good/too bad random choices.
-
$\begingroup$ Thank you very much Kate. I appreciate your reply. Can you please tell me if my random forest is correct? Can it overfit or is it overfitting? How can I find out? Also is it supposed to show better accuracy Than neural net. Thank you again. $\endgroup$– ashDec 3, 2020 at 7:57
-
$\begingroup$ Yes, the random forest can overfit, especially if you pass custom parameters. For overfitting, check accuracies on the train and validation set. If accuracy on the train set is significantly smaller than on the test set, then the random forest is overfitting. $\endgroup$ Dec 3, 2020 at 18:13
-
$\begingroup$ Also, the "No free lunch" theorem says that there is no model universally better than others. Therefore, in some cases, the random forest performs better, in some cases, the neural net outperforms. P.S. If the random forest were always better than the neural networks, why would people use neural networks? $\endgroup$ Dec 3, 2020 at 18:15
-
$\begingroup$ Thank you very much very much for you detailed reply.it really helped. My train accuracy for random forest is (99%) higher than the test accuracy . So I think the random forest is working well. I have one more question. How am I supposed to find out which algo will work best for a particular data? Do I have to test algorithms to find out or is there any certain ways that I can say beforehand that this particular algo will work better for a certain dataset. Extremely sorry for so much questions as I am a noob here. $\endgroup$– ashDec 4, 2020 at 0:32
-
$\begingroup$ No worries. 1. If model works well, check how consistent it performs by using cross-validation. See the following page for examples: scikit-learn.org/stable/modules/generated/… $\endgroup$ Dec 4, 2020 at 2:02