From what percentage should I remove a column that has empty values?

Question

I have a problem. I am currently cleaning and preprocessing my data.

Suppose I have the following data set

ID    CarName            MaxSpeed_in_kmh          Costs        Liter_per_100kmh
1     Mercedes-Benz                  190          100000                   5.5
2     Mercedes-Benz                  180                                   4.9
3     Porsche                        
4     Mercedes-Benz                                                        3.5
5     Porsche                        260
6     BMW                            140
7     VW                             110           50000
8     VW                              90                                  11.8
9     Porsche                        280
10    

Column              Percentage_of_missing_columns
CarName             10%
MaxSpeed_in_kmh     20%
Costs               80%
Liter_per_100kmh    40%

At what percentage is it recommended to completely delete a particular column? For example, I have defined 65% from experience, since it is often hard to fill up more than 65% of empty values (with average, etc.).

Is there a generally valid statement?

According to my example, the following columns would be removed: Costs, because they have more than 65% empty values. The remaining empty columns would be filled.

Or does it vary from data set to data set, if so how do I know what would be a good percentage?

Answer 1

There is no general rule to that - just different ways to handle/impute missing values.

You could remove the column, impute the column (mean/median, k-nearest-neighbor imputation, ...) and additionally flag the imputed columns (flag for 'original' information and 'imputed' information) in order to retain as much original information as possible, and so on [...]. Personally, I like to try the last mentioned approach since I don't simply have to discard potentially valuable information.

However, the only way is to actually try different approaches. It highly depends on the data at hand.