We have a dataset with multiple features ~400 where all of the features have a histogram as you can see in the following picture (sampled only a few)
We thought that this looked like some categorical data where few of the times had a special negative number and other times were zero.
So we treated each feature as category of whether the value is zero or not and we created a new column with binary values, 0 or 1. We did the same thing for whether the value is this special negative number or not. We ended up with one more column.
In other words we did the same thing as if we had three categories for each attribute A, B and C and we wanted to encode them with one-hot-encoding. Where here the A is 0, B is our special negative value and C is anything else. Recall that in one hot encoding you throw away one of the three columns because it is redundant.
Experiments to test our assumption
First experiment: Linear Regression with only original data
Second test: Linear Regression with the binarized features (2 columns for each feature, we had twice as much features)
Third test: Linear Regression with the binarized features plus the original (2 columns for each feature plus the original results is three times as much features)
The results was that the first experiment performed worse than the rest which was expected.
If our theory is correct that these can be represented as categories then the second experiment would perform better in terms of RMSE. And indeed the second experiment performs better.
The surprising result was that when combining the initial data with our binarized ones (augmented dataset with three times more features than the original one) the RMSE was even lower!
Why is that?