LSTM Multivariate time series forecasting with multiple inputs for each time step

Question

I want to predict an output variable for the next day, for each of the users in my dataset. I was thinking of using LSTMs for achieving this.

The dataset

The dataset I am using has multiple inputs for each time step and it is dependent on the value from one of the inputs.

Each user_id has multiple features for each day and gives an output in the range of [1, 10]. Not all users have data for each day.

-----------------------------------------------------------------
|     day    |   user_id   | feature_1 |   feature_n  |  output |
-----------------------------------------------------------------
| 2020-02-11 |      1      |     20    |       9      |    8    |
| 2020-02-11 |      2      |     15    |       8      |    4    |
| 2020-02-12 |      1      |     10    |       2      |    6    |
| 2020-02-13 |      2      |     16    |       9      |    5    |
| 2020-02-13 |      3      |     19    |       1      |    7    |
| 2020-02-13 |      1      |     14    |       4      |    9    |

I want to forecast the output for each of the users for the next day.

I was thinking to split the dataset in multiple datasets for each of the users and use only the data for the user to predict the output, the problem is that my dataset isn't big.

Is there any way to use the whole dataset and after training the model to get as output the output value for each of the users for the next day?

Answer 1

The only possible way is to create a dataset separately for each user; in the end, if you have 10 users, then you would have 10 different unrelated time series in the same .csv, since each user can exhibit specific characteristics. Evidently we cannot expect to throw 10 different unrelated time series into an LSTM and expect decent results.

The solution to this is to extract the users with the most entries (you could start with the user with the most entries) and apply in the first instance a simpler algorithm, not necessarily an ml-based one but a statistical one, such as VAR.

Answer 2

If you want to use RNNs, you start setting an input_length parameter. Then you imagine to slide a window of that size on each trend, and at each step you take an observation for your RNN.

You'd end up with an array of shape:

( # of observations , input length , number of variables )

And the output, of course, is the 1D array of values, output column.

If it's a regression-like problem you can put a single output node. You can either put no activation (i.e. linear activation), or scale the output in [0,1] and use Sigmoid.

That said, the efficacy of an LSTM Neural Network depends from how much data you can train it on. With small datasets, Deep Learning is not very useful and canonical statistical techniques could win against that. What is the size of your dataset?

Answer 3

I have kind of a similar dataset. I'll tell you what I did. I haven't got great results, but it may be due to my data being bad more than anything.

Like @Leevo mentioned, you need to create a sliding window. You can do this with a simple for loop:

def split_sequences(df, n_timesteps_in, n_timesteps_out):
    X, y = list(), list()
    for i in range(len(df)):
        # find the end of this pattern
        end_ix = i + n_timesteps_in
        out_end_ix = end_ix + n_timesteps_out
        # check if we are beyond the dataset
        if end_ix + 1 > len(df):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = (
            df.iloc[i:end_ix][features].to_numpy(),
            df.iloc[end_ix:out_end_ix][features].to_numpy(),
        )
        X.append(seq_x)
        y.append(seq_y)
    return np.array(X), np.array(y)

Now the addition I made which might handle your situation: rather than getting sliding windows for the entire dataset, group your data by user_id, and get the sliding windows for each separate user_id. This way you are sure each sample feature array corresponds with a relevant target array:

def split_sequences(df, n_timesteps_in, n_timesteps_out):
    X, y = list(), list()
    groups = df.groupby("user_id")
    for _, group in groups:
        for i in range(len(group)):
            # find the end of this pattern
            end_ix = i + n_timesteps_in
            out_end_ix = end_ix + n_timesteps_out
            # check if we are beyond the dataset
            if end_ix + 1 > len(group):
                break
            # gather input and output parts of the pattern
            seq_x, seq_y = (
                group.iloc[i:end_ix][features].to_numpy(),
                group.iloc[end_ix:out_end_ix][features].to_numpy(),
            X.append(seq_x)
            y.append(seq_y)
    return np.array(X), np.array(y)