2
$\begingroup$

I'm running a classifier (logistic regression). The information on my dataset are the following:

dataset size= 279 observations 

(80/20 rule)

train size= 233
test size = 56

# of events in train = 31
# of events in test = 8

I think my classifier and results may be affected due to this not equal proportion. Is there any way to avoid bias issues and improve accuracy? What do you personally think of such data?

$\endgroup$
2
  • $\begingroup$ stats.stackexchange.com/a/453377 , refer here $\endgroup$ – Aymuos Aug 31 '20 at 2:34
  • $\begingroup$ The event proportion isn't identical between the train and test set, but it actually can't get any closer with any other split - I'm not seeing much of a problem at all here. $\endgroup$ – Nuclear Hoagie Aug 31 '20 at 15:48
4
$\begingroup$

If you're referring to the fact that your dataset is small:

If you're referring to the class imbalance being 31:202 in train and 8:48 in test:

  • Use AUROC and PRC to eliminate bias in thresholding
  • Also see MCC
$\endgroup$
2
  • $\begingroup$ thank you so much for the advice. Yes, I was referring to the class imbalance in train and in test. I do not know if it makes sense to run a classifier with such imbalanced data, but unfortunately this is the only dataset I have. Should I apply k-fold cross validation and/or AUROC and PRC before building the classifier? $\endgroup$ – Val Aug 31 '20 at 0:27
  • 1
    $\begingroup$ By 'build the classifier' I assume you mean to train the model? Basically with k-fold cross validation, you'd end up training k models, each with a different section of the data. Model 1 is trained on folds 1,2,...k-1, model 2 is trained on 2,3,...k, model 3 is trained on 3,4,...k,1, and so on. Thus they each get 1 fold to evaluate on. Wikipedia would have a better explanation though $\endgroup$ – Benji Albert Aug 31 '20 at 0:36
1
$\begingroup$

I think in case of such unsymmetric data, where the output is outnumbered by one of the classes. Recall would be a good choice of measure than accuracy. The recall gives us the percentage of the relevant class actually predicted by the model.

$\endgroup$
1
$\begingroup$

To complete @BenjiAlbert answer, in case of imbalanced dataset, it is also recommended to use stratified k-fold to preserve the relative class frequencies in each fold. You can find more details in the sklearn user guide here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.