# Regression in Python with many NaN values spread across all columns

I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank values:

value     age     education     gender
32.3                            Male
31.8                            Female
32.8              High school
33.8              Technical school
16.3     18 - 24
35.2     25 - 34
35.5     35 - 44


I converted categorical data by using dummy variables which resulted in below column examples. I guess that the quality of my model will be affected because I have only a single 1 by row and the rest is all 0.

value   18 - 24   25 - 34   35 - 44   College   High school
32.8       0         0         0        0          1
26.4       0         0         0        1          0
16.5       1         0         0        0          0


So my question is, what is the best way to clean and convert the data for given source data structure?

• It depends on the model you use for prediction. Boosting trees usually allow Nan values. Xgboost and lightGBM should be able to work with nan values efficiently without any preprocessing. For other methods there are lots of variants beginning with replace with zero. – keiv.fly Dec 9 '18 at 2:06
• Would you be allowed to use inferences such as if education = High School, then Age = 13-18 (or whatever the age for the region you are considering)? This might not be the best idea if you want to incorporate outliers, but since you are using ranges for the age column, I don't think this data is anyway accounting for outliers and so should not be a problem. – rahs Dec 9 '18 at 2:32