We have data with objects, each of which has a series of events. A series is 1-50 events, revealed over time (a few months). These objects have events come in at different times during a season, so they are not synchronous or regular intervals in any way. We wish to classify each object good or bad based on the information we have at the time, often before the full series of events is revealed. The dataset is fairly small (thousands of rows, hundreds of "good" objects).
I cannot be more specific about the exact nature of the data.
I am looking for ideas of how to best set up model training and evaluation for a model that will work at any time, when some objects have only partial sequences revealed. Of the two, I'm more interested in proper evaluation. If we get evaluation right, we'll be able to try different modeling techniques without lying to ourselves.
As an example, for an object O with events A-B-C-D-E and label L, I thought about putting five different rows in the training (or test) data: (A, L), (A-B, L), (A-B-C, L), (A-B-C-D, L), and (A-B-C-D-E, L). Two possible problems with that idea: 1) those five rows will be correlated (seems biased), 2) objects with more events will get more representation.
I'm not fixed on this idea, it's just an example of what sort of considerations I'm thinking about.
Currently, we are training different models for different time periods, but that feels suboptimal to me. It's hard to manage models, the time periods are coarse, it seems like that process throws away data. (Maybe it's the best we can do, tho?)
I found this survey (Z Xing et al.) on sequence classification, which mentions the problem of "early classification", but doesn't give much detail on how to approach the modeling and measurement problems.
Neural networks are popular, but I think we don't have enough data. I'm fine with translating the sequence to a set of features, and using something more basic (a random forest, logistic regression, whatever). I'm just mulling over the process of how to train and measure one model that can predict for an object no matter how many events are revealed.
Thoughts?
EDIT: This paper (Mori et al.) looks to be highly related.
EDIT 2: There are two timelines for "early" events: 1) the number of events a particular object has, 2) the overall time of year. I'm not sure how to think about separating these two. I think Mori et al. ignore #2 and focus on #1. Perhaps I should do the same.
