0
$\begingroup$

I have a dataset with a categorical column that contains three categories. One of the categories represents 98% of the data, while the remaining 2% are distributed between the other two categories, with a few (maybe around 50) in each. It is worth mentioning that the output for these 50 rows is the same, which suggests that these data points may be important.

However, the data is obviously imbalanced, and I am unable to perform any analysis. Should I drop the entire column, or perform a chi-square test on the data as-is?

$\endgroup$
5
  • $\begingroup$ Please be a bit more specific about what task you try to perform. "Analyzing data" is quite unspecific and can mean a lot of things. $\endgroup$
    – Broele
    Commented Jun 25, 2023 at 20:53
  • $\begingroup$ What do you find problematic about the imbalance? Note that imbalance usually isn’t a problem. $\endgroup$
    – Dave
    Commented Jun 25, 2023 at 21:10
  • $\begingroup$ Hi, @Broele I changed the title to be more specific $\endgroup$
    – user151171
    Commented Jun 27, 2023 at 10:36
  • $\begingroup$ Hi @Dave, I just thought this extreme imbalance would affect Chi-square test, that's my point (is it OK to determine the significance of the column given that I have only 50 out of 14k that belong to the second category?), I mean it's like having a column with almost one category, making it useless. $\endgroup$
    – user151171
    Commented Jun 27, 2023 at 10:39
  • $\begingroup$ Thanks @Dave, this resource helps $\endgroup$
    – user151171
    Commented Jun 27, 2023 at 10:40

1 Answer 1

1
$\begingroup$

To answer your question:

  • First do some analysis over it. Like chi-square test.
  • Try to create a model like RandomForest model, so that you can draw the feature importance, and then you can see how important the column really it.
  • Now if the model suggests that it's not that important then you can drop the column
  • Or if the model suggests that it's an important feature, then try to combine it with other features using groupby or any other techniques. So that you drop the column by keeping its important in other much important column.

Hope I answered your question.

$\endgroup$
2
  • $\begingroup$ Hi @Harshad, My point here is chi-square test viable even if the second category is just 50 out 14k. Wouldn't the significance be affected by that? And thanks for RandomForest suggestion. $\endgroup$
    – user151171
    Commented Jun 27, 2023 at 10:58
  • $\begingroup$ I am not sure if it will be a good approach to go with chi-square test but you can try that one. But I am sure that the RandomForest approach could give you a confident answer. $\endgroup$ Commented Jun 27, 2023 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.