3
$\begingroup$

My initial thought was a neural network but I don't see how a neural network can properly predict interaction between variables (ie. x1 * x2) since each node is just a sum of previous inputs?

Would a decision tree be better suited at capturing the interaction between variables?

My dataset is large, with 400 features and 5,000,000 instances. All data is in percentile and the label is also a percentile. The dataset is quite noisy as well, (customer data, predicting likelihood of becoming a return customer).

$\endgroup$
1
  • 4
    $\begingroup$ There is no generalization that can answer which model will perform best as this is entirely dependent on the specific set of data that you have and the use case. Always try out simpler models first as adding more complex models does not always perform better! $\endgroup$ – Ethan Dec 23 '20 at 4:28
3
$\begingroup$

Probabilistic Random Forest tends to work better then other algorithms on noisy datasets. But the data you are using also plays a major role on whether a algorithm will work or not. Check this paper Probabilistic Random Forest for more details. Happy Learning

$\endgroup$
1
  • $\begingroup$ What about dealing with interactions between features, which I suspect is important but have no idea which features will interact. $\endgroup$ – krissy_fong Dec 23 '20 at 10:54
3
$\begingroup$

Ensemble methods, boosting or bagging, often give predictive accuracies superior to other methods. From my personal experience, I find GBM (ie. Gradient Boosting Regressor over Decision Trees) and LightGBM(faster) often give very accurate predictions.

Check out this diagram on choosing the right estimator.

$\endgroup$
3
  • 1
    $\begingroup$ What about dealing with interactions between features, which I suspect is important but have no idea which features will interact. $\endgroup$ – krissy_fong Dec 23 '20 at 10:54
  • $\begingroup$ Decision Trees GBM are very flexible and should do. Just make sure you control overfitting. $\endgroup$ – Chong Lip Phang Dec 23 '20 at 11:57
  • 1
    $\begingroup$ I am having problems with overfitting haha: stats.stackexchange.com/questions/502080/… $\endgroup$ – krissy_fong Dec 23 '20 at 12:05
0
$\begingroup$

I would make the following models:

  1. a null baseline model
  2. a linear regression model with the most highly correlated features
  3. create polynomial features and do feature selection to just pick the top 10 or 20 features and try those with a linear regression model.
  4. #3 but with ridge regression
  5. a LightGBM model with the original features
  6. If you think you can still squeeze out some performance and it's worth the time/effort tradeoff, move to neural nets. As long as you have a few layers and a decent number of nodes and a non-linear transformation (e.g. RELU) it should be able to pick up interactions.

If something looks promising, go that direction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.