0
$\begingroup$

I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how can I adapt it to my dataset. In their guide they load the dataset and create a sequence:

TIME_STEPS = 288

# Generated training sequences for use in the model.
def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i : (i + time_steps)])
    return np.stack(output)


x_train = create_sequences(df_training_value.values)
print("Training input shape: ", x_train.shape)

The reason why they are using 288 as TIME_STEPS is because they have a value for every 5 mins for 14 days. My questions are:

  1. Does this method split the data or is he just creating a 3D variable in the correct format for the Convolutional Autoencoder?

  2. I have a dataset where there are stored the measurements of 30 devices. Each device has about 4000 values and it is structured as well:

     device01 0.02;0.13;1.15;0.10;8.30;........;4.20
     device02 0.06;0.13;1.40;0.03;7.40;........;6.30
     ........
     device30 0.03;0.24;1.10;0.43;4.40;........;2.30
    

    Since I do not have a timestamp reference in my dataset, how can I define the TIME_STEPS variable?

$\endgroup$
3
  • $\begingroup$ The sequence creation there has an implicit "hop" (how much a new window moves forward from the next) of 1 - which is the maximum one can have. If this creates too many sequences, one can usually increase the hop quite a bit - aka reducing the "overlap" between windows $\endgroup$ – Jon Nordby Mar 24 at 16:24
  • $\begingroup$ Is each of the 4000 values for each of your devices a time-series, sampled at regular intervals? $\endgroup$ – Jon Nordby Mar 24 at 16:25
  • $\begingroup$ @JonNordby Yes, each of those values is sampled every 3 seconds and it's the same for all the devices. $\endgroup$ – Fabio Mar 24 at 17:15
1
$\begingroup$

The length of TIME_STEPS in one sequence is more of a Hyperparameter. That you should try to optimize.

Does this method split the data or is he just creating a 3D variable in the correct format for the Convolutional Autoencoder?

It simply create dataset for a 1-dimsnional convolutional network.Something like this,

$\hspace{3cm}$enter image description here

Each row is an instance. Convolution will happen across the row.

I have a dataset where there are stored the measurements of 30 devices

You may simply assume each data point as a Time-step assuming it was collected simultaneously.
But your data has an additional dimension i.e. Devices.

  • So you should either build one model for each device or take the average of each device and consider it as the single time-step.
  • Or you can apply a 1-D CNN with a 2-D kernel in your first layer, assuming each device as a channel. Check an example here.
  • In case, Keras doesn't allow a 2-D kernel, then use a 2D-CNN with kernel size "30xM". With that, the convolution will happen in only one direction.

    Beware that if you use the last 2 options, then your pre-processing function will have to change i.e each time-step will have "30x1" as a dimension.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.