0
$\begingroup$

I want to incrementally train my pre-trained autoencoder model on data being received every minute. Based on this thread, successive calls to model.fit will incrementally train the model. However, the reconstruction error and overall accuracy of my model seems to be getting worse than what it initially was. The code looks something like this:

autoencoder = load_pretrained_model()

try:
   while True:
      data = collect_new_data()
      autoencoder = train_model(data) # Invokes autoencoder.fit()
      time.sleep(60)
except KeyboardInterrupt:
   download_model(autoencoder)
   sys.exit(0)

The mean reconstruction error when my autoencoder was initally trained was 0.0206 but after incrementally training the model for 30 minutes it has become 0.3737

$\endgroup$
2
  • $\begingroup$ Have you verified that the data distribution is identical between pretrain and training? What happens if you just train online, without pretraining? $\endgroup$
    – Jon Nordby
    Mar 21, 2022 at 8:19
  • $\begingroup$ The data distribution is identical between pretrain and training. I can't only train while receiving this continuous stream of data since the model will not be trained with enough data initially to give an accurate reconstruction error. $\endgroup$
    – sj2000
    Mar 21, 2022 at 12:10

1 Answer 1

0
$\begingroup$

From your question is not clear how you split your data for model performance evaluation. It seems to me that your are only evaluating reconstruction error on trainset.

If I am right it is totally normal that mean reconstruction error is increasing, indeed adding online data your moving towards right on the blu curve below.classical learning curves plot

What you should do is to define a validation method to tune your model and than check the online metrics on the test set. If the model is well trained you will see your training error increasing and your test error decrease.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.