I have a dataset with 50 patients. The patients are tracked over many years with a set of a few thousand features measured at somewhat random intervals. I am trying to predict a certain outcome (this is a regression problem) which is measured either once, twice, or three times per patient over the entire time period the patient is tracked. So far, I have been assuming that each outcome feature is a data point in my Y vector, and using the closest time point where the features are measured for the patient. I want my final model to be able to predict the outcome variable from only one measure of features, which is why I am OK ignoring the temporal aspect of the data. However, I am not sure how to control for the fact that some patients contribute 2 or even 3 samples to my training data. Wouldn't this affect the performance of my model since there is some in-patient variation? How do I account for this?
-
$\begingroup$ Welcome to the site! Ensure that a patient does not appear in both training and test sets. $\endgroup$– EmreCommented Jul 9, 2018 at 20:31
2 Answers
Like all models, the final outcome will depend heavily on the data you use to train.
Given you don't have many samples for each patient and that the samples are temporally irregular, I imagine you might be lumping all recorded data (from all patients) into a single training set... ? In that case, it will be important to either perform random sampling or something like stratified sampling - both without replacement - to make sure patients don't appear in both train and test sets. Stratification simply means to create training/test splits, which do their best to preserve class balance. So if you select one feature as most important, stratification will make sure that training and test sets each have a proportional number of samples exhibiting that feature.
If you are using R, look into the Caret package - here is an example of usage. That is a very powerful package in general and worth investigating (if you use R or not!). If you are using Python, have a look at the Splitter Classes, which include: Stratified K-folds and Stratified Shuffle Split, among others.
If temporal ordering does play a big role, taking the most recent sample is usually considered best practice. For example, measurements driven by general information flow, e.g. finance, would also consider the most recent measurement as the most valueable for prediction.
This is the case in many applications, but perhaps not with your measurements - in medical data, behavioural/environmental conditions of the patient around the time of measuring may outweigh the recentness of the data.
One more observation, in the you are trying to predict some feature that is very specific to a single patient (and you say you may only have one observation from each patient!), then you might have a hard time making accurate predictions.
It depends on your data. You may want to keep only the last sample for each patient. Or you may want to add a column with an integer value that shows the sample count for each patient.
