1
$\begingroup$

I have a dataset with ~6M points, 9 features and two classes. The minority class represents just under 2% of the data. The data is first divided into 100 batches and a different classifier is trained for each batch using the rest of the data.

So far I have tried using sklearn's Naive Bayes classifier trained first using all the data, then downsampling, and then SMOTE. Every time I get this weird shape on the PR curve. I am now trying with random forest, but I am not confident. Does this suggest that the features design is bad? Any suggestions on what would be my next step?

enter image description here

enter image description here

enter image description here

enter image description here

$\endgroup$
  • 1
    $\begingroup$ I’m concerned that in the PCA visual both classes are stacked right on top of each other. Since your input features are only 9, I suggest running some box plots by class to see if there is any differences between the univariate distributions. $\endgroup$ – Jon Aug 22 '19 at 18:49
  • $\begingroup$ I thought the same thing, but since I didn't design the features and only recalculated them on data I obtained from an other lab I wanted to try to reproduce what they did. I don't have access to PR or ROC curve or any other metric beyond accuracy. $\endgroup$ – Seb Aug 23 '19 at 1:33
  • $\begingroup$ It looks like the features all look alike between the two classes. Depending how the data is obtained, sampled, or generated there may be more informative features to consider. Also, when you build your ML model, are you scaling and standardizing the data? $\endgroup$ – Jon Aug 23 '19 at 2:13
  • $\begingroup$ The data is from expensive mass spectrometry experiments and domain experts tell me that those features should be the most informative. Although I agree with you that the boxplots say otherwise. The preprocessing involves only binning in 1000 equally sized bins. Should I try normalizing in some way? $\endgroup$ – Seb Aug 23 '19 at 12:18
  • $\begingroup$ Yes, try normalizing your data $\endgroup$ – Jon Aug 23 '19 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.