I need to implement a deep learning algorithm to predict an ordinal value, called 'Entity', using longitudinal health records data. I read a few articles and guides but I couldn't find a clear explanation or example on how to organize input data; the only thing that I've understood is that I need to use an LSTM node which is designed exactly for this kind of problem. To clarify let me bring an example, let's say that I have this table:
ID | N1 | C1 | B1 | Out |
---|---|---|---|---|
2 | 25 | 3 | 0 | 1 |
2 | 32 | 4 | 1 | 2 |
2 | 52 | 1 | 0 | 1 |
3 | 12 | 2 | 1 | 2 |
3 | 56 | 4 | 0 | 1 |
3 | 73 | 3 | 1 | 3 |
3 | 12 | 2 | 1 | 2 |
The table contains variables of different kinds like numerical, cardinal, and boolean; I need to teach my algorithm to predict the future output with longitudinal data (for example if I input the first two rows with ID=2 the output should be 1 which is in the third row with ID=2).
The rows with the same ID are different data from the same person, checked at different times.
I've understood that LSTM are layers useful when there is the need to catch time relations and the input is in shape (samples, timesteps, features); given that i have few doubts about how to treat and model the training data:
- My dataset is not homogeneous in the timesteps dimension (persons may have different number of rows), but I think it's ok to have tensors with different timesteps length, is that correct?
- if the answer at the previous question is yes, can i train my network with tensors of growing dimension of the same patient? For istance(notice that i want to predict the future Out):
train1 x:[25, 3, 0] -> label:2;
train2 x:[[25, 3, 0],[32, 4, 1]] -> label:1;
BONUS) Since is a complicated problem I was even wondering if I am looking at it in the wrong way and I should treat it like a kinda-regression problem like the one in that example https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
To give a different perspective of the problem let's say doctors classify Alice by looking at the current situation and at previous visits, I would like to implement this behaviour in an algorithm so that when I have Bob current situation and clinical history I can make a prediction for him too based on the "rules" that i learned from Alice examples. The alternative behaviour, that I expressed in the BONUS question, is if I want to reproduce the learning pattern of a Doctor which look at a graph like the one in the figure below and learn how to predict the trend of the y value in another patient based on the "rules" learned in Alice's graph. I'm implementing that on Python using Keras, but I am stuck in this preprocessing phase and I can't proceed.