I have a dataset with the following specifications:
- Training dataset with 193,176 samples with 2,821 positives
- Test Dataset with 82,887 samples with 673 positives
- There are 10 features.
I want to perform a binary classification (0 or 1). The issue I am facing is that the data is very unbalanced. After normalization and scaling the data along with some feature engineering and using a couple of different algorithms, these are the best results I could achieve:
mean square error : 0.00804710026904 Confusion matrix : [[82214 667] [ 0 6]]
i.e only 6 correct positive hits. This is using logistic regression. Here are the various things I tried with this:
- Different algorithms like RandomForest, DecisionTree, SVM
- Changing parameters value to call the function
- Some intuition based feature engineering to include compounded features
Now, my questions are:
- What can I do to improve the number of positive hits ?
- How can one determine if there is an overfit in such a case ? ( I have tried plotting etc. )
- At what point could one conclude if maybe this is the best possible fit I could have? ( which seems sad considering only 6 hits out of 673 )
- Is there a way I could make the positive sample instances weigh more so the pattern recognition improves leading to more hits ?
- Which graphical plots could help detect outliers or some intuition about which pattern would fit the best?
I am using the scikit-learn library with Python and all implementations are library functions.
Here are the results with a few other algorithms:
Random Forest Classifier(n_estimators=100)
[[82211 667] [ 3 6]]
[[78611 635] [ 3603 38]]