I have a fairly large (>100K rows) dataset with multiple (daily) measurements per individual, for a few thousand individuals. The number of measurements per individual vary, and there are many null values (that is, one row may have missing values for certain variables/measurements, but not for all). I also have a daily outcome (extrapolated, but let's assume it's fair to do so, so there is a binary outcome for each day when measurements are taken).

My question goal is to model the outcome, such that I can predict daily outcomes for new individuals.

My background is in research, and I am familiar with some statistics and ML, and overall still fairly new to data science. I am wondering if there are any particular known ML algorithms that can be used to model such data. I am cautious about using logistic regression from something like python's scikit learn because the observations are not independent (they are highly correlated on an individual level). From my knowledge, these kind of data are well-suited for a mixed effects logistic regression or longitudinal logistic regression. However, I haven't been able to find any widely used ML algorithms for it, and I would like to pursue an ML approach rather than fitting a statistical model using something like lme4 package in R.

Could someone recommend an available ML algorithm to model such data?

PS: I did some research and found a few research articles on the topic but nothing widely used or clearly implemented. The structure of the data I am working with strikes me as very common, so I thought I'd ask.


Assuming we are not talking about a time series and also assuming unseen data you want to make a prediction on could include individuals not currently present in your data set, your best bet is to restructure your data first.

What you want to do is predict daily outcome Y from X1...Xn predictors which I understand to be measurements taken. A normal approach here would be to fit a RandomForest or boosting model which, yes would be based on a logistical regressor.

However you point out that simply assuming each case is independent is incorrect because outcomes are highly dependent on the individual measured. If this is the case then we need to add the attributes describing the individual as additional predictors.

So this:

id | day | measurement1 | measurement2 | ... | outcome
A  | Mon | 1            | 0            | 1   | 1
B  | Mon | 0            | 1            | 0   | 0

becomes this:

id | age | gender | day | measurement1 | measurement2 | ... | outcome
A  | 34  | male   | Mon | 1            | 0            | 1   | 1
B  | 28  | female | Mon | 0            | 1            | 0   | 0

By including the attributes of each individual we can use each daily measurement as a single case in training the model because we assume that the correlation between the intraindividual outcomes can be explained by the attributes (i.e. individuals with similar age, gender, other attributes that are domain appropriate should have the same outcome bias).

If you do not have any attributes about the individuals besides their measurements then you can also safely ignore those because your model will have to predict an outcome on unseen data without knowing anything about the individual. That the prediction could be improved because we know individuals bias the outcome does not matter because the data simply isn't there.

You have to understand that prediction tasks are different than other statistical work, the only thing we care about is the properly validated performance of the prediction model. If you can get a model that is good enough by ignoring individuals than you are a-okay and if your model sucks you need more data.

If on the other hand you only want to predict outcomes for individuals ALREADY IN YOUR TRAINING SET the problem becomes even easier to solve. Simply add the individual identifier as a predictor variable.

To sum it up, unless you have a time series, you should be okay to use any ML classification model like RandomForest or boosting models even if they are based on normal logistical regressions. However you might have to restructure your data a bit.

  • $\begingroup$ Thank you for your answer! It makes sense to me. And the assumption you started with is correct. I do not think this is a classic time series problem, since I am not modeling a continuous outcome where the outcome itself is also a predictor (which is my basic understanding of time series). The measurements, however, have a time stamp, of course, varying across a range of time. But other information is more static, like the attributes you describe. $\endgroup$ – Georgiy Apr 30 '20 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.