I've got data representing a sequence of events at irregular intervals and an outcome. There are a fixed number of event types, about 10 of them. The outcome being modeled is binary.

Arguably the following are important to incorporate into the model:

1) Which events occur

2) The order in which the events occur

3) The time lag between events

I.e: User has event A at start, 5 months later event C, and outcome is positive. Another user has event B at start, 3 days later event A, 2 days later event D, and outcome is no.

I've also got some static (non sequence data) for each user which I'd like to incorporate.

What is the best approach to model this?

Thanks in advance.


There are multiple ways to tackle this. I'll suggest two here.

[1] The one that probably requires the least amount of preprocessing work is to throw the event data into a recurrent neural net that can handle variable sequence lengths. Map the event categories to a small embedding space and run it through the RNN. You can then concatenate the remaining static features and put a feed-forward classifier on top. This would take care of 1) and 2). In order to account for 3) there are several tricks.

  1. You can append the time to event to the embedding vectors
  2. You can multiply the embedding vectors with a "time mask" (i.e. feed the sequence of event lagging times into a RNN with sigmoid activation und multiply element-wise. The idea is that the sigmoid activation will put a weight on the event based on the time passed, e.g. an event many months ago will be unimportant and therefore receive an activation close to 0). See section 3.1. in this paper
  3. That paper also suggests another solution where you can learn a joint embedding of time and event (section 3.2). Note that both methods report only minor (but consistent) gains over standard RNNs.

[2] Given that your number of event categories is low you could also just one-hot encode the event or use a tree-based method like gradient boosting and feed the events, the time between events and your remaining features as separate inputs into the model. Unless your dataset is very huge this will perform equally well if not better because you don't have to learn embeddings and have a lot less model parameters.

The only difficulty is that you will now have to fix the sequence length. But you can pad shorter sequences with N/A or 0 values and I would not expect any performance decrease because of it!

  • $\begingroup$ Thank you for the thoughtful answer $\endgroup$
    – cosmosa
    Feb 4 '20 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.