Why augmenting the training data with binary attributes works better for our dataset?

Question

We have a dataset with multiple features ~400 where all of the features have a histogram as you can see in the following picture (sampled only a few)

Our assumption

We thought that this looked like some categorical data where few of the times had a special negative number and other times were zero.

So we treated each feature as category of whether the value is zero or not and we created a new column with binary values, 0 or 1. We did the same thing for whether the value is this special negative number or not. We ended up with one more column.

In other words we did the same thing as if we had three categories for each attribute A, B and C and we wanted to encode them with one-hot-encoding. Where here the A is 0, B is our special negative value and C is anything else. Recall that in one hot encoding you throw away one of the three columns because it is redundant.

Experiments to test our assumption

First experiment: Linear Regression with only original data

Second test: Linear Regression with the binarized features (2 columns for each feature, we had twice as much features)

Third test: Linear Regression with the binarized features plus the original (2 columns for each feature plus the original results is three times as much features)

Results

The results was that the first experiment performed worse than the rest which was expected.

If our theory is correct that these can be represented as categories then the second experiment would perform better in terms of RMSE. And indeed the second experiment performs better.

The surprising result was that when combining the initial data with our binarized ones (augmented dataset with three times more features than the original one) the RMSE was even lower!
Why is that?

Answer 1

Your third model has the most capacity of the three. When you have the values $x$ with special negative value $\xi$ and indicators $\mathbf{1}_0$ and $\mathbf{1}_{\xi}$, the linear model can fit $$ \alpha x + \beta \mathbf1_0 + \gamma \mathbf1_{\xi}$$ When $x>0$ you just get $\alpha x$, when $x=0$ you just get $\beta$, and when $x=\xi$ you get $\alpha \xi + \gamma$; so your model can get the correct (linear) signal out of the positive values and arbitrary values for the two special values.

Answer 2

With the information provided is difficult to draw conclusions. But theoretically speaking this is absolutelly possible. There is what in Data Science is called the curse of dimensionality wich can explain these phenomena (I mean decreasing performance while increasing the number of features). In Machine Learning, the Hughes phenomenon shows that

...With a fixed design pattern sample, recognition accuracy can first increase as the number of measurements made on a pattern increases, but decay with measurement complexity higher than some optimum value.

Maybe this will give you some intuitions or point some aspects of your experimental setting that might be linked to the lower results.

Hope this helps!