class distribution looks like thisthis is how the dataset looks likeI have a dataset with size ~ 500k entries. There are 2 columns, 'product description' and 'level 1'. I am developing my model such that it learns from a training set of 350k and based on the product description for test data, it gives the values in 'Level 1'. A simple linear classifier gives an accuracy of 85% which is too low, I am aiming for 97% atleast. I think this might be because the dataset is imbalanced, the level 1 values in the training data are imbalanced. How do I resolve this? Can I make the upsampling minority and downsampling majority work here?

  • $\begingroup$ Could you edit your post to include the class distribution to see the extent of class imbalance and the number of classes you are using for this task? $\endgroup$
    – shepan6
    Jul 28, 2020 at 6:46
  • $\begingroup$ @shepan6 i dont understand your question...im new to python..could you check the edited question once again if it makes sense to you? thanks! $\endgroup$
    – huy
    Jul 28, 2020 at 7:11
  • $\begingroup$ To print the class distribution for one column, you can use df[column].value_counts() $\endgroup$
    – qmeeus
    Jul 28, 2020 at 8:14
  • $\begingroup$ So what I mean by class distribution is the number of examples you have for each each class. @qmeeus has provided a way to get these values. You will need to visualise them by adding .plt(kind = “bar”) (link to visualise data from pandas dataframes queirozf.com/entries/…) $\endgroup$
    – shepan6
    Jul 28, 2020 at 9:31
  • $\begingroup$ @qmeeus,thanks for the help! $\endgroup$
    – huy
    Jul 28, 2020 at 11:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.