Incremental learning on Autoencoder for anomaly detection

Question

I want to incrementally train my pre-trained autoencoder model on data being received every minute. Based on this thread, successive calls to model.fit will incrementally train the model. However, the reconstruction error and overall accuracy of my model seems to be getting worse than what it initially was. The code looks something like this:

autoencoder = load_pretrained_model()

try:
   while True:
      data = collect_new_data()
      autoencoder = train_model(data) # Invokes autoencoder.fit()
      time.sleep(60)
except KeyboardInterrupt:
   download_model(autoencoder)
   sys.exit(0)

The mean reconstruction error when my autoencoder was initally trained was 0.0206 but after incrementally training the model for 30 minutes it has become 0.3737

Answer 1

From your question is not clear how you split your data for model performance evaluation. It seems to me that your are only evaluating reconstruction error on trainset.

If I am right it is totally normal that mean reconstruction error is increasing, indeed adding online data your moving towards right on the blu curve below.

What you should do is to define a validation method to tune your model and than check the online metrics on the test set. If the model is well trained you will see your training error increasing and your test error decrease.