Time-series prediction: Model & data assumptions in AI/ML models vs conventional models

Question

I was wondering if there was a good paper out there that informs about model and data assumptions in AI/ML approaches.

For example, if you look at Time Series Modelling (Estimation or Prediction) with Linear models or (G)ARCH/ARMA processes, there are a lot of data assumptions that have to be satisfied to meet the underlying model assumptions:

Linear Regression

• No Autocorrelation in your observations, often when dealt with level data (--> ACF)
• Stationarity (Unit-Roots --> Spurious Regressions)
• Homoscedasticity
• Assumptions about error term distribution "Normaldist" (mean = 0, and some finite variance) etc.

Autoregressive Models

• stationarity
• squared error autocorrelation
• ...

When dealing with ML/AI approaches, it feels like you can throw whatever you like as an input (my subjective perception). You are satisfied with the result as long as some prediction/estimation error measurement is good enough (similar to a high, but often misleading R²).

What assumptions have to be satisfied for an RNN, CNN or LSTM model that find application in time-series prediction?

Any thoughts?

ADDED

• Good Article describing my question/thoughts.
• Medium Article discussing model assumptions + tests, but not in the context of more advanced models
• I read the 100-page ML Book- Unfortunately almost no content about model assumptions or how to test for them.

Answer 1

Great question @Maeaex1 !

First of all why do we even need assumptions in models (generally speaking )?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources

---

---

---

But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

So to conclude: there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.

Answer 2

I would like to comment, but i can't with my reputation. So here is an [article] (https://arxiv.org/pdf/1802.04474.pdf) with some assumption. You can treat neural network in the way of this article (non parametric estimation) and do the same assumption than this field (where there is few assumption, except about regularity of the distribution).

Hope it helps.