0
$\begingroup$

I have a problem. I am currently cleaning and preprocessing my data.

Suppose I have the following data set

ID    CarName            MaxSpeed_in_kmh          Costs        Liter_per_100kmh
1     Mercedes-Benz                  190          100000                   5.5
2     Mercedes-Benz                  180                                   4.9
3     Porsche                        
4     Mercedes-Benz                                                        3.5
5     Porsche                        260
6     BMW                            140
7     VW                             110           50000
8     VW                              90                                  11.8
9     Porsche                        280
10    
Column              Percentage_of_missing_columns
CarName             10%
MaxSpeed_in_kmh     20%
Costs               80%
Liter_per_100kmh    40%

At what percentage is it recommended to completely delete a particular column? For example, I have defined 65% from experience, since it is often hard to fill up more than 65% of empty values (with average, etc.).

Is there a generally valid statement?

According to my example, the following columns would be removed: Costs, because they have more than 65% empty values. The remaining empty columns would be filled.

Or does it vary from data set to data set, if so how do I know what would be a good percentage?

$\endgroup$

1 Answer 1

0
$\begingroup$

There is no general rule to that - just different ways to handle/impute missing values.

You could remove the column, impute the column (mean/median, k-nearest-neighbor imputation, ...) and additionally flag the imputed columns (flag for 'original' information and 'imputed' information) in order to retain as much original information as possible, and so on [...]. Personally, I like to try the last mentioned approach since I don't simply have to discard potentially valuable information.

However, the only way is to actually try different approaches. It highly depends on the data at hand.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.