-1
$\begingroup$

I have a data.set that contain around 3000 observations. Every observation falls in one of the five categories (these are pre-defined).

I am using randomForest and gbm from the h2o package for classification. Before I run the algorithms i optimize the hyperparameters of each algorithm using the h2o.grid.

For model validation I am having a look at the confussion matrix for both algorithms, for both in-sample and out-of-sample.

The issue is that in sample the algorithms perform relatively well for all 5 categories but out of sample they they only perform well for one category. I guess that is very common.

I have 2 explanations: the 5 categories are not equally divided, this means that for a couple categories I have a few observations. Second, is that the algorithms are overfitting.

The "1m dollar question" is how can someone avoid overfitting ?

I thought of running a preliminary analysis on my data set, to see which variables (in my case most of them dummy variables) in my data set are the "most important" for the classification, and then use those, instead of the whole data set. This could be tested by taking the means for the variables per category and which variable has the highest difference across the categories. Could this be a good idea ?

I understand that the subject is too broad, but I think some ideas from experienced people on this topic could be useful for everyone !

$\endgroup$

2 Answers 2

1
$\begingroup$
  1. Try solving the imbalance problem by using something like SMOTE to make each class roughly proportional.
  2. Make sure you are splitting your in sample and out of sample data randomly. Sometimes people will take the first x% and call that training, but if your data has any order to it, the sample may not be representative.

I would not try removing variables from the data set until you've addressed the above.

$\endgroup$
4
  • $\begingroup$ I had already taken care of these 2 matters. I check the proportions of each category in my train set and my test set and are quite the same, plus i split my data randomly. $\endgroup$
    – quant
    May 23, 2017 at 7:50
  • $\begingroup$ From the other comments you left, you are still too imbalanced. With one class being 42% and another being 1/10 that at 4%, I am not surprised that it is ignoring some classes. Try getting them within a couple %. $\endgroup$
    – CalZ
    May 23, 2017 at 13:55
  • $\begingroup$ the h2o.randomForest can take as parameter balance_classes which is Logical. Balance training data class counts via over/under-sampling (for imbalanced data). Defaults to FALSE. Is this the same with SMOTE ? $\endgroup$
    – quant
    May 24, 2017 at 8:19
  • $\begingroup$ SMOTE is a much more advanced version of over/under-sampling which usually just means deleting records or duplicating records. It wouldn't hurt to try though. $\endgroup$
    – CalZ
    May 24, 2017 at 12:06
1
$\begingroup$

You will need to provide the exact distribution of the 5 levels so I can comment if its a case of overfitting. Let's say one of your class is 90% of the data, and the model is giving 90% accuracy on out of sample data (100% for the majority class) and 0% for rest of the classes. This is NOT a case of overfitting, the best the model could come up with was to put all in one class, it gave decent accuracy too. In general to avoid overfitting crossvalidation is really important. I suggest using mlr package rather than h20 because the crossvalidation there is easier. As suggested by CalZ you can use SMOTE to make your training classes in better proportion(but remember to not over/under sample the out of sample data). As far as your feature selection technique goes,You can use chi-squared tests or information gain to plot the top features, also there are many advanced feature selection algorithms like Boruta etc. You can use them, but remember don't blindly trust these algorithms, they're not magic. Only do feature selection if lets say you have 100 variables( for 3000 datapoints). For 20-30 features, you can apply the model on all of them, plot the importance plot for gbm and then try to remove the less important variable and see if the accuracy increases.

$\endgroup$
3
  • $\begingroup$ The proportions (in the test set) are as follows: 16%, 4%, 14%, 42%, 24% which is more or less the case also in the train set. After the classification the proportions are 16%, <1%, 5%, 63%, 15%, for each category respectively $\endgroup$
    – quant
    May 23, 2017 at 7:59
  • $\begingroup$ Your result is pretty balanced but there is still a decreasing trend in your data. What I can infer from the result is that the classes having less proportion are getting neglected by model a bit. Try applying more weights to classes in less proportion, it should help. $\endgroup$ May 23, 2017 at 8:42
  • $\begingroup$ the h2o.randomForest can take as parameter balance_classes which is Logical. Balance training data class counts via over/under-sampling (for imbalanced data). Defaults to FALSE. Is this the same with SMOTE ? $\endgroup$
    – quant
    May 24, 2017 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.