I've got a collection of historic log files for a process that repeats daily, roughly the same each day. I want an NN that, from the moment that the daily process starts, estimates what time the whole process will finally complete. As the daily process progresses, the estimate should become more accurate and converge on the actual completion time as soon as possible. A degree of confidence in the prediction would be useful too.
The daily process is a collection of mostly serial subprocesses. Mostly serial because some subprocesses have no impact on the end time ( critical path vs non ). There are other pattens hidden in the data. Weekends take longer than weekdays. In the real world, maybe others too. There is a small amount of noise in the data, subprocess duration can be affected by completely external factors. The performance of one given day has little / no expected correlation with the next day, although there may be long term trends.
I've had a little success with a simple RNN but results have been underwhelming, even after trying a lot of hyperparameters. Best results so far have been from one-hot encoding and "cross joining" to expand each day into multiple samples and padding them to the same maximum dimensions...
Xsample1.1 = [Event1, ] Y = EventN Xsample1.2 = [Event1, Event2, ] Y = EventN Xsample1.N = [Event1, ..., EventN ] Y = EventN
I've encoded weekday as one-hot variables and normalized datetimes to one universal day in preprocessing.
All the examples I find are for predicting the next N samples, sliding forward, not a fixed end sample. I'm using ipython3, pandas and keras and I've generated a test population using some simple logic I'd expect an NN to easily handle. YHat for each file is the last datetime. The .debug files capture how the prediction should evolve but that information is not available from the training data and so is not used in the NN.
Is this a time series or regression problem, what type of NN should I use and how should i structure the input data for best results.