0
$\begingroup$

I've spent a few days on this and am starting to think I'm missing the obvious solution as this doesn't seem like a very uncommon problem.

As an example dataset: I have 100 measurements with each a different constant speed as target and a length of 10000 data points. Every measurement has additional 30 features that are temperatures, airflow, humidity, etc. My goal is to predict the speed depending on the features/conditions.

My initial dataset so far has the shape (100, 10000, 30) as in (amount of measurements, length of measurements, amount of features) which I would convert with a sliding window shape. All the solutions I have found so far create one big dataset which (in my understanding) would mean that the independent measurements are treated as one single measurement.

As the hard transitions between the measurements harm my predictions: How can I train the network without creating one big measurement that includes these 'transition mistakes'? I want to use a larger part for training and evaluation and then test a few of the remaining measurements on their own. Therefore I presume my input_shape has to fit a single measurement (10000, 30) and the output target as speed has (10000, 1). How do I transform my data into the window shape while keeping the measurements separate and insert it into my network for training/testing?

I've tried using the batch_input_shape but couldn't find a solution how to incorporate this with the sliding window and existing input_shape so I'm not sure if this could even solve my problem. (I've also thought about training a model each per measurement and trying to ensemble them but that seems way too complicated for this kind of problem.)

$\endgroup$
2
  • $\begingroup$ Are these 100 measurements independent i.e. any 80 of these can be picked as train sets? $\endgroup$ – 10xAI Feb 15 at 8:13
  • $\begingroup$ @10xAI Yes, every single one is based on a different speed and environment status. There is no sequence in which they must be used or dependence in between the measurements. $\endgroup$ – Emyn Arnen Feb 15 at 9:10
0
$\begingroup$

I think what you're looking for is to change your dataset into a supervised learning problem. Keras has its own TimeSeriesGenerator but that wont be of use as it only has a single step output. Someone created an alternate version of the TimeSeriesGenerator. Using this, you can specify the inputs and outputs per sample to change your data into a supervised learning problem which will then be able to fit seamlessly in your LSTM network.

Hope that helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.