Regarding imbalanced multilabel dataset

Question

I am doing multilabel news classification in python language.The dataset I have has two files. First CSV contains articles at each row. Second CSV contains corresponding labels to each article. Here is the snapshot of label file

Here is the dropbox link....https://www.dropbox.com/s/7huzh41je735oqn/labelset.csv?dl=0

1. Is the dataset imbalanced one?
2. How do I distribute this dataset properly into training set, validation and test set?

Note: By properly I mean, Can this unbalanced dataset be split into proper proportions in training,validation and testing set?

Answer 1

I'll go through your questions one by one:

Is the dataset imbalanced one?

The distinction between a balanced and an imbalanced dataset is a matter of degree. If you think that the difference in number of observations for each class is too big, I suggest you to train your model using Mini-Batch Gradient Descent, and build any mini batch in a way that observations have the same frequency. This would attribute greater weights to the less frequent observations.

How do I distribute this dataset properly into training set, validation and test set?

Make sure that all the classes are evenly distributed among the sets.

Answer 2

Yes, the dataset you have been working with is an imbalanced dataset but this problem could easily be tackled if rather than predicting the outcome directly, you will tackle this problem by top 3 predictions. Over the top 3 predictions, you can also add one more check for the threshold to validate the predictions and finalize the prediction.

Apart from this, I would suggest you to manually split the dataset into training and validation set in order to maintain a consistency in the data and help tackle co-variate shift problem.