I'm trying to build a predictive model from about 1 million rows of data. My goal is to predict a certain numerical value.

I have the intuition that I should use very few numerical binary columns so I don't get data points that are too separated, a.k.a., the curse of dimensionality. Is this true? Besides, is it the same for numeric columns?

  • 2
    $\begingroup$ What do you mean with "sparse data points"? Please explain also why it would be a problem. $\endgroup$ – Paul May 26 '19 at 14:42

Did you try it?

In general, I believe the curse of dimensionality is way overrated. As a rule of thumb, the curse of dimensionality refers to having more columns than rows. I doubt that you have 1M columns...

A robust algorithm, such as Logistic Regression, can often deal with sparse data. I sometimes get the feeling that this algorithm doesn't get the credit it deserves. Especially in production (due to high bias and low variance), I prefer it whenever possible. Nothing worse than an algorithm that fails in production...

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.