0
$\begingroup$

When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another.

Some sources say, columns with missing values should be dropped when the percentage of missing values is more than 5-10%, other sources say the threshold is 25%, 50%, 80-85%, etc.

It is also said that null value columns should be only dropped when the number of records is in millions.

In general, under what circumstances should a column with missing values be dropped, with regards to the quantity of missing values & size of dataset? Is there a clear cut answer to my query, or should the problem purely be solved based on intuition & understanding of the dataset?

NOTE: I only need clarity on the dropping of columns with null values, not imputation of these missing values. Please answer accordingly.

$\endgroup$
3
  • 4
    $\begingroup$ In my experience there's not really a hard and fast rule, it really depends on how important the columns are to your predictions. In other words, try dropping and see how well things work vs. imputation and other methods. $\endgroup$ Nov 6, 2023 at 15:32
  • $\begingroup$ Would you like to share more about the missing values column that you are looking to drop or keep. $\endgroup$
    – Kriti
    Nov 6, 2023 at 17:08
  • $\begingroup$ I'm not working on any particular dataset. I want to know in general, how the decision to drop a column with null values is made. $\endgroup$
    – Apoorva
    Nov 6, 2023 at 17:46

1 Answer 1

3
$\begingroup$

In general, dropping a column (feature) or not does not depend on how many missing values there are, but is this column useful for your task.

Sometimes we do feature selection to drop columns with no missing value (dropping because they are irrelevant); sometimes we keep a column even though 99.9% missing (say these missing values have meaning, or give useful info once combined with other features).

On the other hand, we may have different decisions on a same feature given different problem, with or without missing data.

So back to square one, always focus on the problem (the why), not the technique (the how). When in doubt, do experiments to figure out. It is the 'science' bit that counts.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.