0
$\begingroup$

I have a dataset with 5 features (excluding the date) [Result, Ward, Age, Facility, Resource] . The train dataset has non-anomalous data, and the test dataset will have some anomalous data. This dataset is used to train unsupervised LSTM for anomaly detection. The idea is that the LSTM should be able to reconstruct the 'Result'. If the actual 'Result' in test dataset differs too much than the reconstruction, it should categorize it as anomaly.

When creating sequences, I am not sure if I should include all the 5 features? The following is the sample code I used to create sequences. For now, I kept all the features for X, and only 'Result' for y.

I think putting only 'Result' in y_train make sense because I want the model to learn the pattern from other features and produce the 'Result'.

TIME_STEPS = 30

def create_sequences(X, y, time_steps=TIME_STEPS):
    Xs, ys = [], []
    for i in range(len(X) - time_steps):
        X_seq = X.iloc[i:(i + time_steps)].values
        y_seq = y.iloc[i + time_steps]
        Xs.append(X_seq)
        ys.append(y_seq)
    return np.array(Xs), np.array(ys)

# For training data
X_train, y_train = create_sequences(train_df[['Facility', 'Ward', 'Age', 'Service resource', 'Result']], train_df['Result'])

# For testing data
X_test, y_test = create_sequences(test_df[['Facility', 'Ward', 'Age', 'Service resource', 'Result']], test_df['Result'])

When printing, you will see:

X Training shape: (45082, 30, 5)
X Testing shape: (19549, 30, 5)
y Training shape: (45082,)
y Testing shape: (19549,)
$\endgroup$
1
  • $\begingroup$ If it's a forecasting task, you could include yesterday's result with the other 4 features, all of which are used to forecast today's result. If, however, the data/task is not structured as such, then your approach sounds right: using the 4 features only, try predicting the corresponding result. $\endgroup$ Commented May 20 at 10:46

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.