Binary classification: best ways to pre-procees the data

Question

About the dataset I have a training dataset of

• 129 columns(last column being the classes, i.e., y values)
• 6068 rows

I have to train some algo to do binary classification. The data set has

• 701 examples of type A
• 5367 examples of type B

The test set consists of 1398 examples. Here is the accuracy I got for various algorithms.

• voting ensemble -> 0.73963
• stochastic gradient boosting -> 0.77682
• Adaboost -> 0.75107
• bagging classifier(Decision tree) -> 0.76538
• Random Forests -> 0.75250
• Extra trees -> 0.75393

All the above results are from kaggle so they probably are just from half the test set. The above methods were implemented using scikit-learn library in python

Could someone please suggest ways to improve accuracy, may be by things like dimensionality reduction or better algorithms. Also, please provide sample code in possible.

Answer 1

A better clarity about the kind of problem that you are trying to solve will be really useful.

• I any case the first thing I will suggest, is please go through your features and try to understand the relationship with the response. All tree based methods are highly prone to overfitting and hence they don't generalise well. If you come to the conclusion that some variables show a linear relationship and some show a non-linear then you might be better off using Linear methods such as LogisticRegression or LinearDiscriminantClassifier(if your features show an indication that they are sampled from a normal distribution). Obviously you can use PCA or regularization methods if you are certain that some features are highly correlated and offer no prediction value.

• Secondly, if you have done all the above then you are probably done with the feature selection step. The next approach is feature creation. See if certain variables are non-linear in nature. Then try adding higher degree polynomial terms. Try diagnosing interaction effects between features, try to include features for those interactions. If there are any features you think are redundant to the problem. Think again. See if you can extract any information out of them.

Kindly tag the link of the problem here.

Answer 2

There are two steps to this problem: feature selection/ dimensionality reduction and selecting the predictive model. Selecting the 'best' features to use in the model will often improve the accuracy, and there are quite a few methods you can use.

• When you are working with continuous data, you can use a regularization method such as Lasso or ElasticNet to select the best features for the model. These are both available in sklearn, but they do require some parameter tuning to find the right hyperparameters to get the best results.
• You can also take a look at sklearn.feature_selection. These are more statistical approaches to select features, generaly based on their p-values. This also only works with continuous data.
• Another approach is calculating the Gini importance from decision tree models, such as RandomForest. This is useful when you have continuous and categorical data. Here is an example: http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html

After you have selected the best features, you want to choose the right model for binary classification. The go-to model in this case is logistic regression. There are multiple hyperparameters in sklearn.linear_model.LogisticRegression and in order to get the best results, you may have to perform some grid searches to find the right parameters.

Here are some additional resources that may prove helpful:

• http://blog.nycdatascience.com/student-works/bnp-claims/
• http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-problem-abhishek-thakur/
• http://insidebigdata.com/2014/10/22/ask-data-scientist-bias-vs-variance-tradeoff/

I added the last link, discussing bias vs. variance tradeoff, because I think it is important to understand when testing your models.