I have a
data.set that contain around 3000 observations. Every observation falls in one of the five categories (these are pre-defined).
I am using
gbm from the
h2o package for classification. Before I run the algorithms i optimize the
hyperparameters of each algorithm using the
For model validation I am having a look at the
confussion matrix for both algorithms, for both
The issue is that in sample the algorithms perform relatively well for all 5 categories but out of sample they they only perform well for one category. I guess that is very common.
I have 2 explanations: the 5 categories are not equally divided, this means that for a couple categories I have a few observations. Second, is that the algorithms are overfitting.
The "1m dollar question" is how can someone avoid overfitting ?
I thought of running a preliminary analysis on my data set, to see which variables (in my case most of them dummy variables) in my data set are the "most important" for the classification, and then use those, instead of the whole data set. This could be tested by taking the means for the variables per category and which variable has the highest difference across the categories. Could this be a good idea ?
I understand that the subject is too broad, but I think some ideas from experienced people on this topic could be useful for everyone !