I am starting as a PhD student and we want to find appropriate materials (with certain qualities) from basic chemical properties like charge, etc. There are a lot of models and datasets in similar works, but since our work is pretty novel, we have to make and test each data sample ourselves. This makes the data acquisition very very slow and very expensive. Our estimated samples will be 10-15 samples for some time, until we can expand it.

Now I want to use this samples to make a basic predictive model, but with as much 'good generalization' as possible. I will use this model to screen other possible candidates from a large pool of properties to find best probable materials, and will then proceed to make them for testing.

Now I clearly don't expect performances anywhere near 95% or so, but I want a working model with predictive capability that will actually help me to find some of the best probable material candidates, so we can expand our work. I am uncertain if I can (or rather should) use some of regular ML methods like dataset splitting and cross-validation. So I would appreciate your thoughts.

Since our data size is minuscule, I have been searching for ways to improve its robustness. These our my ideas:

1- Use an ensemble model to avoid overfitting and avoid skewed biases (using algos like elasticnet, SVM, random forests, etc).

2- Setting heavy regularization to avoid certain biases that can arise from small data.

3- Using algos that arrive at the minimum periphery faster.

I would appreciate any suggestions on how I can improve this model as much as possible, to reach the best generalization performance.

I have also thought about synthetic data generation a lot. Do you have any suggestions on how I can go about it?

  • $\begingroup$ how many features (aka $x$-variables)? How does correlation look like between the $x$ $\endgroup$
    – Peter
    May 13, 2020 at 15:30
  • $\begingroup$ We're still testing, so I dont have the full data yet, but we are measuring 5 properties. I intend to do feature engineering and selection by converting (poly, log etc.) and multiplying them to find the best correlated combinations, but I think 3 or 4 final features will be good for the model and will avoid some of the overfitting. $\endgroup$
    – Sadegh
    May 13, 2020 at 16:11
  • $\begingroup$ Is $y$ continuous or binary/multiclass? $\endgroup$
    – Peter
    May 13, 2020 at 16:21
  • $\begingroup$ y is continuous and I will be using regression models for that. $\endgroup$
    – Sadegh
    May 13, 2020 at 16:41
  • $\begingroup$ bootstrap to get a range of values then calculate the confidence intervals with the newly found values. $\endgroup$
    – mccurcio
    May 13, 2020 at 19:11

1 Answer 1


From what you say, I think you should start with checking three options:

I) Ordinary least squares (OLS): Just run a „normal“ linear regression. This will not yield great predictions, but you could view the model as a causal one, if you can assume a linear relation between $y$ and $x$. When you have five predictors and 35 observations, you have a total of 29 degrees of freedom which is „okay“. When you estimate the model in „levels“, so just values as they are, you can directly interprete the estimated coefficients as marginal effects. E.g. a model $y=\beta_0+\beta_1 x + u$, tells you that when $x$ increases by one unit, $y$ changes by $\beta_1$ units, just like a linear function.

II) You can use Lasso/Ridge/Elastic Net: All of them are linear-like models with a penalty term to „shrink“ $x$ variables if they are „not useful“. This works like automatic feature selection if you like to say so. There is a great package by Hastie et al. for R. You can find it here. It is also available for Python.

III) Maybe (!) boosting could be an option as well: You would (likely) need to do some feature selection/engineering on your own. But Boosting is able to work with a small number of observations, with highly correlated features, and it often works well with highly non-linear problems. There are LightGBM or Catboost as possible Python packages. Find some minimal examples here.

With II) and III) you will find that you are not really able to „set aside“ a number of observations to check if your models work (because you don’t have much data). You could use cross validation (Ch. 5 in ISL, link below), but you need to see how it works. Instead of going for a predictive model, I tend to say that you might be better off starting with a „causal-like“ OLS model. With OLS you do not really need a „test-set“. OLS is very robust.

Since you seem to be new to statistical modeling, you might benefit from having a look at „Introduction to Statistical Learning“ (Chapters 3 and 6 in particular). The PDF is online and there is code for the Labs in Python and R. The advanced book would be „Elements of Statistical Learning“.

Good luck with your project!

  • $\begingroup$ Thank you for your help. What is the difference between predictive and casual like models that you mentioned? Is it just the evaluation methods? $\endgroup$
    – Sadegh
    May 18, 2020 at 9:40
  • $\begingroup$ There is no difference wrt to the math, its more about the general way to approach a problem: Causal modeling will only work with linear models to directly make a model interpretation from the coefficients (e.g. in econometrics) and you need a "theory" how the data generating priocess looks like. Predictive modeling will use whatever model works but will generally use a "test set" to demonstrate the accuracy of the chosen model. $\endgroup$
    – Peter
    May 18, 2020 at 10:34
  • $\begingroup$ Here is one of the most prominent examples what econometrics looks like jrvargas.files.wordpress.com/2011/01/… $\endgroup$
    – Peter
    May 18, 2020 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.