1
$\begingroup$

i'm still new in machine learning. currently i'm creating an anomaly detection for flight data. it is a multivariate time series data that include timestamp, latitude, longitude, velocity and altitude of the aircraft. i'm splitting the data into train and test with 80% ratio. i used the keras LSTM autoencoder to do a anomaly detection. so here's my code

def create_sequence(data, time_step = None):
    Xs = []
    
    for i in range (len(data) - time_step):
        Xs.append(data[i:(i + time_step)])

    return np.array(Xs)

# pre-process to split the data

dfXscaled, scalerX = scaledf(df, normaltype=normalization)
num_train = int(df.shape[0]*ratio)

values_dataset = dfXscaled.values

train = values_dataset[:num_train, :]
test = values_dataset[num_train:, :]

# sequence input data [sample, time step, features]
train_input = create_sequence(train, time_step = time_step) 
test_input = create_sequence(test, time_step = time_step) 

train_time = index_time.index[:num_train]
test_time = index_time.index[num_train:]

# model 
model_arch = []

last_layer = num_layers - 1
for x in range(num_layers):
    if x == last_layer:
        model_arch.append(tf.keras.layers.LSTM(num_nodes, activation='relu', return_sequences=True, dropout = dropout))
    else:
        model_arch.append(tf.keras.layers.LSTM(num_nodes, activation='relu', input_shape=(time_step, 4), dropout = dropout))  
        model_arch.append(tf.keras.layers.RepeatVector(time_step))
        
model_arch.append(tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(4)))
model = tf.keras.models.Sequential(model_arch)
opt= tf.keras.optimizers.SGD(learning_rate=learning_rate)
model.compile(loss=tf.keras.losses.Huber(),
              optimizer=opt,
              metrics=[tf.keras.metrics.MeanAbsolutePercentageError(name='mape'), tf.keras.metrics.RootMeanSquaredError(name='rmse'), "mae", 'accuracy'])
history = model.fit(train_input, train_input, epochs=epochs, batch_size = num_batch, validation_data=(test_input, test_input), verbose=2, shuffle=False)

when i do a model evaluation, it come up with 100% accuracy model evaluation result

is it good to have 100% accuracy ? or my model is overfitting the data ?

$\endgroup$

1 Answer 1

1
$\begingroup$

Usually indicates something is wrong.

In your case, things which do not seem right:

  1. One can easily get ~100% accuracy in anomaly detection - just keep predicting the majority class.
  2. Is this model really for anomaly detection? Anomaly detection is a classification problem, but your metrics (MAPE, RootMeanSquaredError etc.) are regression metrics.
$\endgroup$
4
  • $\begingroup$ for point 2, could you tell me what metrics for anomaly detection ? i just use someone code to create that model. $\endgroup$
    – farhanrbn
    Commented May 23, 2022 at 6:27
  • $\begingroup$ Anomaly detection is classification, so a metric for classification should be used. Since there is a heavy class imbalance, I suggest you to define your custom cost matrix to balance true/false positive/negative based on business requirements. $\endgroup$
    – lpounng
    Commented May 23, 2022 at 7:10
  • 1
    $\begingroup$ But back to square one, one should never simply copy&paste code, replace the inputs and pray for it to work - ML never works this way. $\endgroup$
    – lpounng
    Commented May 23, 2022 at 7:12
  • $\begingroup$ i see, thank you for your advice $\endgroup$
    – farhanrbn
    Commented May 23, 2022 at 7:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.