# LSTM for capturing multiple patterns

I am trying to use an LSTM to predict daily usage for users. I have data for (say) 90 days of usage for a large number of users. Based on business knowledge (and initial analysis) we know users fall roughly into different categories. E.g. daily users would have a non-zero usage almost every day, weekly users would have one or two days of non-zero usage every 7 days and monthly users would have a couple of days with non-zero usage per 30 days.

Sample data where each column is one day starting from October 1st and each row is data for one user. (The usage 'cycle' of each user might start on any day).

User 1: 10,  8, 10,  9, 0, 0, 11, ...
User 2:  0,  0,  0, 20, 0, 0,  0,  0,  0, 18,  0,  0,  0, ...
User 3: 40,  0,  0,  0, .....


where User 1 might be a "daily" user, User 2 is a "weekly" user and User 3 is a monthly user.

My first question is that can a single LSTM/deep learning model capture these different types of patterns? The goal is to predict the daily usage (next couple of days based on past 90 days) for individual users.

Currently I am using a really simple LSTM (in Keras):

model = Sequential()
model.compile(optimizer='rmsprop', loss='mse')


To help the model 'capture' the fact that different users might have different levels of usage I added the average usage (of non-zero values) for each user as the first data point for each row. The remaining 90 data points for each user remain as shown in the table above.

My second question is if I really need to add the average of the values to 'help' the model?

The problem is that even after 100 epochs the error remains unchanged. And finally what can I change to make this to work?

• First of all, you have to tell us what you're after? Are you trying to classify the three different type of users? If so, LSTM is an overkill. Much simpler methods are available. Second, to account for the level effect, you need to normalize the rows, not putting the mean as the first data point. – horaceT Jan 3 '17 at 23:46
• @horaceT, thanks for the reply. I want to predict daily usage values for next couple of days based on past 90 days for individual users. I have updated the question to reflect this. As for my second question thanks for the suggestion on normalizing the data. – mtariq Jan 4 '17 at 0:00

Modeling is a process where you invent hypotheses and then try to reject them. Without seeing your data, there is no guarantee any of these would go far. But here are some thoughts.

1. 90 days of daily observations seem too few, which won't lead to much confidence no matter what model you use.

1.1 If you treat each user as one sequential observation, yes your dataset is sufficient for models like RNN. But ask yourself the question, are there users out there who are unique and different and you don't know about how they differ from the rest.

1. You seem to start with a belief that there are three types of users. I would suggest don't. Let the data tells you how many types are there. And for the purpose of predicting future usage, you don't care about this type information.

2. Recurrent neural net is a good choice if you believe the user behaviors are temporally dependent individually and have cross-sectional dependence. Otherwise, a time series model like ARIMA may do a better job and require less resources to fit.

3. Lastly, if you want to normalize the data, you should do so by subtracting each row by the row mean and divide by the row standard deviation. By adding the mean as the first observation, you're going to confuse the neural net and it's not smart enough to treat this data point differently. Unless there is a huge variance across users, I would start without normalization, fit a model, figure out some sort of out-of-sample MSE, then repeat on normalized data.

EDIT : There is no reason to train multiple RNNs on this set of data if you do not have a good segmentation argument.

• Thanks for the detailed reply. A few cmts 1. I have 90 days of data for millions of users. Agreed that 90 days is not enough to capture patterns for users who only have a few days with non-zero usage. But would it be enough for other users? 2. Agreed that I should not assume there are only 3 types of usage patterns. But my main question is if a single LSTM can be trained to capture multiple types of patterns? 3. I will definitely look into ARIMA. In my case I need a prediction of no-usage (value of 0) or usage (actual usage value) on daily basis. 4. Point taken about normalization. – mtariq Jan 4 '17 at 3:55
• @mtariq See my edit above. – horaceT Jan 4 '17 at 5:03