Suppose we have a Neural Network (or any machine learning model), and the goal is to perform incremental learning as new data comes in on a regular schedule.

There is extensive literature showing improvement in performance or convergence speeds if the features are normalized to mean 0 and unit variance prior to training, such as Jayalakshmi and Santhakumaran (2011).

However, most feature pre-processing tends to be dependent on the distribution of the features itself. For instance, let $\{X_{t,k}\}$ be a features matrix. In the case of Z-score normalization, we have the transformed matrix $\{Z_{t,k}\}$ where $Z_{t,k} := \frac{X_{t,k}-\mu(X_{k})}{\sigma(X_{k})}$ where $\mu, \sigma$ are the mean and standard deviations of the $k$th column.

In an incremental learning approach, we may have a sequence of batches $\{X_{t,k}\}_{\tau\geq 0}$ so by performing any preprocessing or feature transformations individually on each batch $\tau$, there's the implicit assumption that each batch should be i.i.d. for the model to learn accurately.

The dataset I am working with is a matrix of multi-variate non-stationary time series, so the possibility of concept drift is quite likely. How can I address this in the model?

Some thoughts:

  • Fit the preprocessing only at the beginning $\tau = 0$, and apply that to all subsequent batches. In this case, if concept drift is severe enough or for large enough values of $\tau$, the transformed matrix may no longer have mean 0 and unit variance - which defeats the purpose of preprocessing in the first case and may make the model less robust.
  • Fit the preprocessing for every batch. Then, small differences in the distribution of the features matrix could lead to the model overfitting on that particular batch and not being able to generalize to future batches.
  • Don't do any preprocessing at all. Then due to concept drift, the model may be training on unseen data and also may lead to subpar results in the future.
  • Stationarize the data (i.e. I(k) differentiation). This is a potential solution so that each batch will be roughly i.i.d., but also wipes out memory of past values beyond the order k. I am using an LSTM NN since it can work with non-stationary time series.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.