I am trying to train a model to detect gender in a dataset of CEO speeches. Here are the datasets that I have:
- Final Dataset: 20K CEO voices analyzed (around 95% male)
- Testing dataset (?): 1K CEO voices analyzed from the final dataset, less unbalanced because I added more females on purpose (80% male)
- Training dataset: 6K voices analyzed from audiobooks and TED talks (55% male).
For now, I have been trying different models by training and splitting dataset #3 (70%, 30%). I get good accuracy (95%) using this method. However, when I apply the trained model to dataset #2, I get an accuracy of 85%.
I am not sure what to do. Should I undersample women in the training dataset so that its distribution is more similar to the final dataset?