3
$\begingroup$

I am trying to train a model to detect gender in a dataset of CEO speeches. Here are the datasets that I have:

  1. Final Dataset: 20K CEO voices analyzed (around 95% male)
  2. Testing dataset (?): 1K CEO voices analyzed from the final dataset, less unbalanced because I added more females on purpose (80% male)
  3. Training dataset: 6K voices analyzed from audiobooks and TED talks (55% male).

For now, I have been trying different models by training and splitting dataset #3 (70%, 30%). I get good accuracy (95%) using this method. However, when I apply the trained model to dataset #2, I get an accuracy of 85%.

I am not sure what to do. Should I undersample women in the training dataset so that its distribution is more similar to the final dataset?

$\endgroup$
2
$\begingroup$

Over- or undersampling should be your second choice. Currently, the best method to deal with class-imbalances is to use the weights argument which sklearn- and keras-classifiers support (see for example the DecisionTreeClassifier).

On a general sidenote, I would recommend focusing on the f1-score, AUC, and the confusion matrix to evaluate the model's performance, as accuracy might be not as informative in this case.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much for the advice. $\endgroup$ – user1029296 Jul 23 '19 at 5:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.