0
$\begingroup$

I have a lot of data where im trying to distinguish red data from blue, but all the features look like this basically (see imgur).

https://i.stack.imgur.com/zsYVc.png

They overlap a lot, and I can only get roc auc score of 0.65 .

So far in terms of data-processing ive only used class weights to balance it. I have tried different kinds of dense/dropout networks, with the different features but only 0.65 is the best I can get, most hover around 0.63-0.65, no matter how many layers or epochs I add.

For example a simple DNN/dropout yields 0.65: """ Dense(512, activation='relu'), Dropout(0.3), Dense(512, activation='relu'), Dropout(0.3), Dense(512, activation='relu'), Dropout(0.3), Dense(512, activation='relu'), Dropout(0.3), Dense(512, activation='relu'), Dense(1, activation="sigmoid") """

Some resnet modification I found online also only provided 0.65.

So i have tried CNN and dense layer networks only, so far.

Any hints or ideas I could do to get better AUC scores?

$\endgroup$
6
  • 1
    $\begingroup$ If your data is not able to seperate the classes you can't do much using CNN or any other machine learning technique. So try to create discriminative features first, before working on the CNN architecture. $\endgroup$ – Graph4Me Consultant Sep 8 '20 at 9:54
  • $\begingroup$ @Graph4Me I think CNN are often used to find these "discriminative features" through the backpropagation process without having to find them manually, so using raw data with CNN instead of features with DNN could be a lead $\endgroup$ – etiennedm Sep 8 '20 at 12:00
  • $\begingroup$ @AlexSanteri could you give more details about what you are trying to classify, the context and what you have done so far ? $\endgroup$ – etiennedm Sep 8 '20 at 12:01
  • $\begingroup$ @etiennedm Of course this is true. But as I understood, the input is distributed as shown in the image. In such cases, it is difficult also for a CNN to seperate the classes. If the input (features) of the two classes are too much correlated, there is not much a CNN can do.. $\endgroup$ – Graph4Me Consultant Sep 8 '20 at 14:38
  • $\begingroup$ @Graph4Me If the plot is the input raw data you are right. My understanding was that the plot was one feature distribution. If so, maybe there is information in the raw data that is not extracted by the features. $\endgroup$ – etiennedm Sep 8 '20 at 14:44
0
$\begingroup$

I think the other comments about improved feature engineering and sharing more about your problem statement are correct.

I would just add that one helpful tip when thinking through these kinds of problems could be how you as an expert would classify these points by hand. If you would have trouble manually separating the two classes (which the distribution you have shown would lead us to believe), then almost any ML model will have the same trouble.

Without knowing more I'd say that this is potentially related to the concept of Bayes error rate (the amount of intrinsic error which cannot be further reduced with better modeling). If you flip a fair coin, sometimes you will get heads and sometimes you will get tails. There are no better features which will help you decide how to map that action (flipping the coin) to a value (heads/tails), there will always be some intrinsic error because the process itself is random.

In case you are interested: https://en.wikipedia.org/wiki/Bayes_error_rate

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.