Obtain a model where each feature vector is past few samples and labels are future few samples?

Question

So, I have this data set where each instance is made of past 20 samples of 2 variables. Labels are next five samples. So every instance looks like this:

Instance1_feature = [s1_var1 s2_var1 ... s20_var1 s1_var2 s2_var2 ... s20_var2]
Instance1_label = [s21_var1 s22_var2 ... s25_var5 s21_var2 s22_var2 ... s25_var2]


Instance2_feature = [s2_var1 s3_var1 ... s21_var1 s2_var2 s3_var2... s21_var2]
Instance2_label = [s22_var1 s23_var2 ... s26_var5 s22_var2 s23_var2 ... s26_var2]

All the data are categorical. var1 takes value in {1,2,3,4,5}; var2 takes value in {6,7,8,9,10} I am familiar with machine learning. Hence I am looking for a way to develop a model to predict labels for a new instance. I have reasons to believe that by looking at the past samples, it is possible to predict future samples. Its like stock prices.

Any help, links to code samples will be highly appreciated.

EDIT This is a dataset I collected while driving. I want to see if I can predict next 5 seconds of my driving, given I have past 20 seconds of data. That's why I created the feature vectors and labels like this. The values {1,2,3,4,5} and {6,7,8,9,10} are categorical, as in the variables are split into segments.

Answer 1

I'm not sure why your data set looks like this (having different steps of future result as a single label) so provide some info about the context of your data.

But you can use any Neural Network, Fuzzy Network and so on. you just need to consider your labels as continuous variables and then in final step round you network outputs. You probably don't need any recurrent or dynamic network, because your data already has lots of delays in it which is possibly able to represent dynamic behavior of your system behind data.

You can also use Multi-Step ahead prediction instead. for example using your Instant1_feature predict or estimate [s21_var1 s21_var2]. then use a feature vector which looks like Instant2_feature but with your estimations of [s21_var1 s21_var2] replaced with real values of [s21_var1 s21_var2].

I hope this helps.

Answer 2

This looks like almost an ideal use case for a Recurrent Neural Network (RNN) model. I say almost, as your output variable is 2 separate class vectors, that part is slightly fiddly.

The main advantage of using a RNN is that the model would be built on the assumption that s1_var1 is of the same type as s2_var1 etc. Whilst with other models you could flatten the time series into a 10 x 20 = 200 long vector, and the output likewise into a 50 long vector, and treat the whole problem as a single step supervised learning, such a model would not include the same assumption.

For your problem, the RNN would have 10 inputs (corresponding to the 5 classes of each of 2 variables) and 10 outputs. You would train it by presenting your input sequence one sample at a time in the correct order 1 to 20, then continue running the network 5 more steps, comparing the outputs against expected. When predicting, you do similar, but then just read off the outputs as predictions, one at a time.

There are a few different Python libraries that support RNN models, and there are different internal choices (such as using LSTM layer versus GRU) which you might want to explore. I am currently learning the Keras framework, which supports some options for RNNs. Probably the example code which learns to predict an exponentially-decaying sine wave would be a good place to start trying to understand the Keras model - although that's a regression, it should give you some insight into how RNNs can be built for this kind of series-based data.

---

An alternative to RNN here might be a hidden markov model - if you have some insight into the internal state of the system, that could be a better option.

Answer 3

If you'd like to keep things straight and simple for a start, you may want to view this as a standard categorial classification problem. As a second step, you use multi-step-Prediction, as @eulerleibniz suggested. To render this approach feasible, you only predict the label for the next second (not for five) as a first step.

You may use a random forest or gradient boosted trees algorithm. Both need little parameter tuning, are reasonably well-behaved, and implementations are widely available (e.g., in Python and R). You can have a go at your problem with these algorithms.

In a second step, you then repeat the whole thing, and combine your input data with the prediction just generated to predict the next label. And so forth, until you've got the next five labels.

If you're just a little bit lucky, the results are already good enough. If not, at least they provide a benchmark for more laborious approaches like RNN.