What regression model can handle tiny amounts of data?

I'm trying to use machine learning to predict properties of a material during a crash test, but each data point requires physically crashing an expensive toy car, so I can only gather a few hundred data points. The end goal is to use the trained model to figure out how to optimize material strength given the parameters I can tweak in the material, which I'm using as inputs.

What type of model would be able to:

1. Handle very few data points (minimizing number of crashes necessary)

2. Accurately learn a likely very complex relationship between the inputs (design choices) and the output (performance in crash)

3. Be able to then be explored to figure out, in as few crashes as possible, which changes to the inputs should be made to maximize the output

• what is y (continuous, categorical)? Does 3) mean that you want to make causal statements? Aug 19 '19 at 6:46
• how many x (meaning observed variables)? Aug 19 '19 at 6:47
• @Peter y is continuous, and considering we control nearly all x, we are in a position to make causal statements. We have a few hundred x, though we can drop any that are shown to have minimal influence on y Aug 20 '19 at 2:33

The description of your problem is vague. In case you want to do causal analysis, you need to consider the bias-variance-tradeoff. For a causal analysis you would stick to low bias (e.g. using a "best linear unbiased estimator" like ordinary least square regression), while most ML models, such as neural nets or boosting, go for low variance.

Still not clear to me: how does your data look like. If you have more features/variables ($$X$$) than observations ($$i$$) your problem is high dimensional. For a high dimensional problem, use the Lasso. If $$i$$>$$X$$ and you have say at least 40+ degrees of freedom, you could simply apply OLS regression. If you (can) use OLS and you care for significance and confidence bands, use robust standard errors (hc2, hc3).

As mentioned above, you will likely have high variance using OLS or Lasso. What you can do to work on that in a causal model, is to make linear transformations of $$x_i$$, e.g. add polynomials, or you can add interactions of two $$x$$, e.g. $$x_3=x_1 x_2$$.

Finally, you could use generalized additive models (e.g. with regression splines) to model "complexity" if it originates from non-linearity in $$X$$. However, in this case you go in the direction of decreasing variance (at the cost of probably introducing unwanted bias).

I guess the nature of your problem is of "high dimension". This is a little special (and I'm no expert here). Have a look at "Elements of Statistical Learning", Chapter 18. The book gives a really good overview of possible options to deal with this problem.

As you want to handle very complex relationship between the inputs, model should be strong enough. It seems that neural network would perform better than svm for example. The problem is the number of points in dataset. However, it just means, that you should try to develop appropriate architecture for your task. There are many challenges in wich nn shows good results on small datasets (with a few hundreds of instances). So, I suggest you experiment with architecture. There are two links with examples how to build baseline model https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/ (dataset for regression problem)

https://janakiev.com/notebooks/keras-iris/ (iris is a small but easy dataset for classification)

This step is rather easy. Then you can try to change number of layers and neurons, add dropout to prevent overfitting on small dataset. Although it would demand efforts, results could be high.

Other good model, as for me, is gradient boosting. It includes ensemble of trees (GB usually based on trees) and performs good results on small datasets https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html But this model should also be tuned for your problem, regularization techniques will help to reduce overfitting on a small number of examples.

The main message is: there's not universal algorithm, which works great from scratch with small data and high complexity, but there is a range of tools to make powerful algorithms perform good on small observations (regularization, dropout, depth and number of trees in GB, learning rate).

• Sorry to say this, but your answer is so not helpful in this context since NN do by definition require larg(er) amounts of data and they are not suited for causal analysis as the aim to minimize variance. Boosting is more like it, but still not suited for causal analysis. Aug 20 '19 at 9:37
• @Peter thank you for the notice! Although I like your answer about variance and bias, I wouldn't agree, that nn is inappropriate for small dataset :) Even in unet paper authors mentioned that they deal with small amount of data. It depends of course on many factors. But not always it's possible to reduce dimensionallity using PCA or something like that. And of course you are right, that nn primarily is used with hughe amount of observations.
– Lana
Aug 20 '19 at 16:20
• No worries, Lana, we work in a highly complex field and the question is vague. Your answer still makes a contribution by highlighting several options to approach the problem... Cheers! Aug 20 '19 at 21:55