I have a large dataset that has session length records per user basis. And I am trying to predict the purchase behaviour based on the session length. But this data has multiple zero in the session length column. How should I deal with these zeros? Should I impute the zeros with other values? Kindly suggest.
1 Answer
$\begingroup$
$\endgroup$
The question you have to ask yourself is whether or not having a 0 is really a missing value, or actually a session length. On it's own, it doesn't seem like a problem. Some users could have a really short session. I'm assuming it's on a website or something and they could have loaded the page than immediately left. It also depends on how the data came to you, there might have been some processing beforehand you don't know about. What if some of the values actually are 0, and other values are missing, but have been filled in with 0? You likely won't be able to tell the difference.
But, if it is truly representing missing, from a statistical point of view, it's more conservative to throw out those records, even if you're potentially losing information in other columns. It's only valid to impute values if the data is missing at random. If it is, then there are several options for imputing, the simplest being using the mean/median value.
From a machine learning point of view, it doesn't matter what the missingness mechanism is as much, you just need to impute it with a value to be able to model at all. And since it's already 0, it's ready to go. But if you were to impute it, it would typically be done during variable preprocessing, such as if you want to center the column. After doing that, impute the mean of the centered variables. And there you have several options for centering, normalizing the column, min max scaling, max absolute value scaling, etc.
In some situations, even if you impute values, it may be useful to create an additional variable indicating whether or not the variable was missing to begin with.
There is no simple answer, it's probably worth experimenting with a few options and seeing what works best for the data.
