What is the best way to optimize the parameters in a Sklearn classifier, when I have little data?

What is the best way to optimize the parameters in a Sklearn classifier when I only have a data set with 684 rows and 177 columns, and the column I want to predict has 3 labels?

I know I should split my data into training, validation, and test set, and then find the parameters to train the training set that maximizes a metric in the validation set and use this optimized classifier in the test set. However, when I did this using a decision tree classifier, the parameters that worked best for the validation set showed a worse result in the test set than the default parameters. So what is the best way to find the best parameters in this data set? I do not know if it helps, but my data set is very sparse.

• You could look into bagging and boosting. Also, decision tree classifiers have very high variance. You could use a random forest (which aggregates decision tree classifiers to reduce variance), but I would suggest thinking about using a different model. – Andrew Maurer Aug 8 '19 at 4:14

1 Answer

You can't get around the test/train thing. I guess in your case, the problem is the choice of method. When I read your data description, I suspect that you might achieve better results with regularization, e.g. Lasso. See some examples here.

Basically you would apply a multiclass logit with lasso penalty in your case. Alternatively you could also check boosting with lasso (aka $$l1$$ penalty). Here is some playcode with the iris data.

Since your data has many columns and not so many rows, I might go for lasso (not ridge or elastic net), because lasso can shrink features to zero if they don't make a meaningful contributin to the classification. This is especially helpful with high dimensional data.