# Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?)

I am currently collecting second by second data regarding buyer vs seller initiated trades for different financial instruments (securities mostly). If there are more buyer initiated trades in a given second, then that second's data point would contain a positive value in the pertinent feature. If there are more seller initiated trades, then there would be a negative value. And if either there is an equal amount of buy vs seller initiated trades OR if there are simply not any trades in a given second, there will be a 0 for the feature in that data point. Along with this feature, there are several other features that are based on what occurred in the preceding seconds (eg if the value discussed above was 12 for the data point immediately preceding the current point, then the second feature for the current data point would be 12 - please let me know if this is not clear) After much troubleshooting, I have concluded that if there are too many data points with too many 0's for features, the classifier simply wont work. When I print out the probabilities of evaluation data points falling into different classes, I simply get

0:NaN,1:NaN


for all model evaluation points I try to classify. (I am using logistic regression from apache-mahout. In total have 183 features, but over 40million data points. There are three categories to which the data point can be classified)

I have found that if I set the default value to 1, then I no longer encounter this error, e.g. if there are no trades, the value will be a 1, if there is one seller initiated trade, the value will be 0.

So with all this in mind, I have two related questions:

1) Has anyone else encountered this issue? e.g. if you have a vector with x features, and for a majority of the data points, a majority of the features contain 0's, is this know to give issues?

2) Is shifting all values up by a constant (such as 1) a valid fix to this issue? I assume that if this constant is applied to all values, then it shouldn't skew the data, but I figure it won't hurt to check with the experts.

Also, I'm new to this, so if you believe that my question could use more info please let me know, and if you could give me ideas of what information to include, it would be greatly appreciated.

• What classifier are you using? Jan 11 '15 at 15:43
• logistic regression in apache mahout Jan 11 '15 at 17:28
• The problem is not with zeroes per se but with too many of one class and not enough of the other; logistic regression is usually fitted with maximum likelihood (although I can't speak specifically for the Mahout implementation) and that causes problems for the algorithm. A search for "rare events logistic regression" on the Stats.StackExchange might help you. I'd write a bigger answer but right now I'm posting from my phone and only have a few minutes Jan 11 '15 at 18:31
• Are the vectors where you encounter this error all zeroes? If so, and you're not fitting an intercept/bias term, this is your problem Mar 13 '15 at 13:30
• And you could fix it by simply including a bias Mar 13 '15 at 13:30