Why augmenting the training data with binary attributes works better for our dataset?

We have a dataset with multiple features ~400 where all of the features have a histogram as you can see in the following picture (sampled only a few)

Our assumption

We thought that this looked like some categorical data where few of the times had a special negative number and other times were zero.

So we treated each feature as category of whether the value is zero or not and we created a new column with binary values, 0 or 1. We did the same thing for whether the value is this special negative number or not. We ended up with one more column.

In other words we did the same thing as if we had three categories for each attribute A, B and C and we wanted to encode them with one-hot-encoding. Where here the A is 0, B is our special negative value and C is anything else. Recall that in one hot encoding you throw away one of the three columns because it is redundant.

Experiments to test our assumption

First experiment: Linear Regression with only original data

Second test: Linear Regression with the binarized features (2 columns for each feature, we had twice as much features)

Third test: Linear Regression with the binarized features plus the original (2 columns for each feature plus the original results is three times as much features)

Results

The results was that the first experiment performed worse than the rest which was expected.

If our theory is correct that these can be represented as categories then the second experiment would perform better in terms of RMSE. And indeed the second experiment performs better.

The surprising result was that when combining the initial data with our binarized ones (augmented dataset with three times more features than the original one) the RMSE was even lower!
Why is that?

Your third model has the most capacity of the three. When you have the values $$x$$ with special negative value $$\xi$$ and indicators $$\mathbf{1}_0$$ and $$\mathbf{1}_{\xi}$$, the linear model can fit $$\alpha x + \beta \mathbf1_0 + \gamma \mathbf1_{\xi}$$ When $$x>0$$ you just get $$\alpha x$$, when $$x=0$$ you just get $$\beta$$, and when $$x=\xi$$ you get $$\alpha \xi + \gamma$$; so your model can get the correct (linear) signal out of the positive values and arbitrary values for the two special values.