1
$\begingroup$

My Dataset:

I have data for vehicles - mainly engine sensor data but also gps location, weather etc. The data is high frequency - every second. I have aggregated to 1 minute.

I roughly have somewhere in the order of 300_000 minutes of data per vehicle. (and 100's of vehicles)

What I want to do

I am wanting to forecast the current day using other variables (e.g forecast engine temperature based on engine rpm, vehicle speed, air temperature etc)

This is to detect anomalies in the engine which can then be flagged - e.g engine temperature too high. I will forecast based on a model trained on normal data (for some vehicles I have engine failures) and then compare the forecast against the actual - the larger the difference the greater the anomaly.

The data looks like this enter image description here (picture shows roughly a week), there are an unknown number of data points a day (depends on how long the vehicle was driven - e.g. 1 hour 1 day and then 14 hours the next day).

I have tried using some models, arima/svm etc, and I now am trying an LSTM. A literature review suggested that deep learning was commonly used for similar tasks, particularly lstms/cnn's.

The problem - the model doesn't to be any good. It basically fits a straight line at the avg which slightly bends, as you can see from the picture it should jump up when the engine is under load (I have engine load as an explanatory var) and down when the vehicle is going slow or still (I have how fast the vehicle is moving also) It also stops really early (based on validation), stops on epoch 2 or 3. I think I am probably making a silly error somewhere.

Example of a model I have tried. I used the last 1000 minutes of data to predict the next 500 minutes of data (the average days has about 515 minutes per day) so use last 2 days ish to predict the next day

 tf.keras.Sequential([

    tf.keras.layers.LSTM(32, return_sequences=False),
    tf.keras.layers.Dropout(rate=0.5),
    tf.keras.layers.Dense(500*num_features,
                        kernel_initializer=tf.initializers.zeros()),
    tf.keras.layers.Reshape([500, 10])
])

Questions

  • why does the validation error barely decrease? It will often decrease on epoch 2 and then never again leading to early stopping.
  • what is a sensible batch size? Each day varies, does the batch size 1000 train, 500 forecast make sense? (500 is about the avg per day)
  • Does including time make any sense? The time of day doesn't give an indication of what the vehicle is doing, and I have temperature to control for enviroment.
  • Is the model bad because it is not possible to fit a model for this data? How would I know?
  • Could I pad the data so that each day has the same number of data points? Does that make any sense?
  • I want a sensible basic model I can then build on and improve, does the model above make sense?
  • Does this setup make sense? I want to predict the current day period with the final model. This training predicts the next day, which I think is the same?

Thanks for any help! I am the only data scientist at my company so I have no-one to discuss it with. Thanks

$\endgroup$

1 Answer 1

0
$\begingroup$

I want a sensible basic model I can then build on and improve, does the model above make sense?

I think that's a good call; to start simple and iterate on that. The model makes sense but I think the model size, sequence lengths, and forecasting lengths, make the model prone to convergence issues.

what is a sensible batch size? Each day varies, does the batch size 1000 train, 500 forecast make sense? (500 is about the avg per day)

I'd suggest starting with a smaller batch size in the ballpark of 8. This will give the model more opportunity to update the weights, and the updates will be less likely to land in a local minimum.

Ultimately, batch size will be a hyperparameter to tune, but you can set it to value that works reasonably well and fine-tune it later if needed.

Does including time make any sense? The time of day doesn't give an indication of what the vehicle is doing, and I have temperature to control for enviroment.

Whilst the absolute time might not be that useful, perhaps you could combine it with some other information into 'cumulative hours driven over lifetime of engine'. This feature would track the total number of hours the engine has been in use over all of the sequences available for that car.

I suppose the time of day might have some utility. If you think that engines started at 3AM are going to have a different normality signature to those started at 10AM when it's generally warmer, then maybe it would be worth including the time of day. However, this would not be very effective if you've only got a few outliers at 3AM. So perhaps it's best to exclude the time of day, with the option of working it back in later if desired.

why does the validation error barely decrease? It will often decrease on epoch 2 and then never again leading to early stopping.

The following will likely help:

  • Try a much smaller batch size, 8 for example
  • Use shorter windows, limited to 100 or a few hundred steps
  • Centre and scale the data if it's unscaled
  • Train for longer, so if early stopping is kicking in at the start, disable it.

Could I pad the data so that each day has the same number of data points? Does that make any sense?

I think that would have been a viable option if the sequence lengths were more similar. For the data you have, however, the significant degree of padding required for the shorter sequences might distort the learning process.

LSTMs start to become ineffective beyond few hundred time steps. The data being fed in should be windowed within that limitation. So I think you should run a sliding window over each sequence, and each window becomes a new sample. For example, if an engine has data from t=0 to t=499, then the windows will be 0-99, 100-199, ..., 400-499. This creates 5 'new' samples (windows) out of the original full-length sequence.

Example of a model I have tried.

It seems like you're using a sequence-to-vector architecture, where the model ingests an entire sequence first, and then renders the full forecast in one go.

The model will need to compress the sequence of data into a single representation, and then unpack that into a long multi-step forecast in one shot. For long sequences, the model would become hard to train.

It might be more effective to use a sequence-to-sequence architecture, where the model makes a forecast at each step in the input sequence. This allows for more error gradients to flow through the model and they don't have to flow through time as much.

so use last 2 days ish to predict the next day

Do you need to predict the entirety of the next day in one go? The usual approach with forecasting is to predict the next step(s) given the current and previous steps. Then, when the next time step comes to pass, you again use the current and previous steps to predict the next one; you're always predicting one or a few steps into the future. Doing a 'one-shot' prediction of 500 steps into the future will probably mean the later time-points are inaccurate compared to the earlier ones.

The data is high frequency - every second. I have aggregated to 1 minute.

When you have a working model, it could be worth revisiting this aspect. You might be using mean aggregation, which will smooth things out. Min/max aggregation is another one to consider, in that the clue to an engine's imminent failure could be the extreme swings it makes rather than its one-minute average.

For a well-performing model, you might be able to improve things further by training on the high-resolution data.

I think aggregation is a sensible starting point though, as you've done.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.