I have a dataset with quotes from an insurance company. I am trying to create a model to predict how much should the company charge the customer according to the different variables. Two of the variables are related to a second driver. One of them is driver2_licence_typeand the other one is dirver2_licence_years.

I am interested in known how to deal with missing values on driver2_licence_years in order to perform either a multilinear regression or a decision tree/random forest regression. There are two main cases.

Case 1

When driver2_licence_type is not NaN I thought it was safe to fill driver2_licence_years with average number of years, because we know there is a second driver but we just don't know how much experience the driver has. However, the price is not likely to follow a linear relation with experience, since very old drivers may be charged more due to loss of abilities. However, I don't know the precise effect. Should I instead do a previous analysis on how years of driving experience explains insurance fees and choose an age that gives the average price? Is it better to try to find what sort of functional relation can be drawn between years and price and then transform the variables accordingly?

Case 2

When both variables are NaN we're assuming that there is no second driver. I originally thought of filling driver2_licence_years with zeros, but I am not sure if the effect of having a non-experience driver should be the same as not having a second driver (one could say it is more dangerous having someone with little experience that having no one). Here I am not sure what to do. From the point of view of a decision tree, it maybe be sensible to add another variable that specifies whether there is a second driver or not, and use this to decide whether to look at years or not. Or maybe I should simply have two different models depending on whether there is a second driver or not. What would you suggest in this case? This answer provides some ideas and I have also read about giving it the value -1, but I am still unsure.


1 Answer 1


My suggestion here would be considering different models for single drivers and additional drivers.

Regarding missing values

(1) This would be the case for additional drivers

When driver2_licence_type is not NAN, but driver2_licence_years has missing values; I would consider average of driver2_licence_years on the basis of driver2_licence_type, driver2_licence_years for the data that does not have missing values. To do this if would group by driver2_licence_type and average driver2_licence_years. I would use this dictionary to compute missing values for driver2_licence_years.If still there are missing values ; I would then consider an overall average.

(2) This would be the case for single drivers

When both driver2_licence_type and driver2_licence_years has missing values; I would consider dropping these two columns and not considering them at all.

If you do not wish to create two different models, I would create an additional variable as you mentioned to flag which one's are single driver and which one's are additional driver (say Single/Additional). The missing values for case (1) would remain same and missing values for case (2) for driver2_licence_type would be None/Other; assuming you would be using encoding methods to convert the variable from categorical to ordinal for your data modelling and missing values for driver2_licence_years can be 0.

Your data model will consider the difference between no driver versus a novice driver with 0 yrs experience because of the encoding values for driver2_licence_type for a particular data row.

This is the ordinal example that you can use for converting the categorical column to ordinal - This is the ordinal example that you can use for converting the categorical column to ordinal

You can compute additional variables like as shown below and then consider just numerical variables for your model-

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.