Time series prediction without sliding window

Question

I've got a collection of historic log files for a process that repeats daily, roughly the same each day. I want an NN that, from the moment that the daily process starts, estimates what time the whole process will finally complete. As the daily process progresses, the estimate should become more accurate and converge on the actual completion time as soon as possible. A degree of confidence in the prediction would be useful too.

The daily process is a collection of mostly serial subprocesses. Mostly serial because some subprocesses have no impact on the end time ( critical path vs non ). There are other pattens hidden in the data. Weekends take longer than weekdays. In the real world, maybe others too. There is a small amount of noise in the data, subprocess duration can be affected by completely external factors. The performance of one given day has little / no expected correlation with the next day, although there may be long term trends.

I've had a little success with a simple RNN but results have been underwhelming, even after trying a lot of hyperparameters. Best results so far have been from one-hot encoding and "cross joining" to expand each day into multiple samples and padding them to the same maximum dimensions...

Xsample1.1 = [Event1, ]              Y = EventN
Xsample1.2 = [Event1, Event2, ]      Y = EventN
Xsample1.N = [Event1, ..., EventN ]  Y = EventN

I've encoded weekday as one-hot variables and normalized datetimes to one universal day in preprocessing.

All the examples I find are for predicting the next N samples, sliding forward, not a fixed end sample. I'm using ipython3, pandas and keras and I've generated a test population using some simple logic I'd expect an NN to easily handle. YHat for each file is the last datetime. The .debug files capture how the prediction should evolve but that information is not available from the training data and so is not used in the NN.

Is this a time series or regression problem, what type of NN should I use and how should i structure the input data for best results.

Answer 1

Implementing this with a recurrent neural network is not that difficult. My suggestion is close to what you already suggested but instead of sampling different points in time I would just broadcast your target over all time steps. If there are a total of N time steps, and we want to predict on steps t=1 to t=N-1, our X is a matrix of shape [N-1, k] with k the number of features. Y is a sequence of shape [N-1] with all the values being equal to EventN.

This is similar to what you actually want to achieve, namely at every time step you want to predict the final value. Your loss function will be calculated over all time steps and backpropagated through time. It's possible that you might want to weigh your function as a function of the time step, maybe later predictions are more or less important, you'll need to do some analysis on this. Another recommendation would be to add some feature of time in your feature explicitly, that might help the network.

Answer 2

How big is N ? If it's not a huge number (if it's huge you need to use NN with online learning to achieve same results without inverting a matrix), how about to use simple linear regression and linear algebra?

In short you will have N models for every stage, when new event comes you reevaluate your model

If you will use linear algebra, all that you need is to estimate projection matrix (every line of this matrix is your events chain on step k [1:N]): $$ P=A(A^TA)^{-1}A^T $$