I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These reports are helpful summaries of information for the user to read later and gauge their progress. The data looks something like this.

user  |  hours until first report
3     | 20
4     | NaN
7     | 24
9     | 13
6     | 500

A NaN value likely means that no report was ever written for this user because they unsubscribed before one could be written. I thought to fill in the blank values with the median, most common value, or the mean but this change in the data would show that unsubscribed users received written reports in the same time frame as the average user, which is incorrect.

Instead of filling the missing value with the mean, I am considering adding a feature that tells whether a report was written at all and changing the NaN values to -1 so that I can pass the data to a sklearn algorithm.

user  |  hours until first report  | report written
3     | 20                         | 1
4     | NaN                        | 0

Is this a valid way of solving this particular problem? What are the risks that I should be aware of?

  • $\begingroup$ Why not just replace it with some other number like -1? $\endgroup$ – Nain Dec 21 '17 at 5:27
  • $\begingroup$ So, you are just creating a new boolean feature based on values in another column and you are going to use both in the algorithm? I think it would just be two highly correlated variables. $\endgroup$ – Ankit Seth Jan 19 '18 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.