8
$\begingroup$

I was wondering if there was a good paper out there that informs about model and data assumptions in AI/ML approaches.

For example, if you look at Time Series Modelling (Estimation or Prediction) with Linear models or (G)ARCH/ARMA processes, there are a lot of data assumptions that have to be satisfied to meet the underlying model assumptions:

Linear Regression

  • No Autocorrelation in your observations, often when dealt with level data (--> ACF)
  • Stationarity (Unit-Roots --> Spurious Regressions)
  • Homoscedasticity
  • Assumptions about error term distribution "Normaldist" (mean = 0, and some finite variance) etc.

Autoregressive Models

  • stationarity
  • squared error autocorrelation
  • ...

When dealing with ML/AI approaches, it feels like you can throw whatever you like as an input (my subjective perception). You are satisfied with the result as long as some prediction/estimation error measurement is good enough (similar to a high, but often misleading R²).

What assumptions have to be satisfied for an RNN, CNN or LSTM model that find application in time-series prediction?

Any thoughts?

ADDED

  • Good Article describing my question/thoughts.
  • Medium Article discussing model assumptions + tests, but not in the context of more advanced models
  • I read the 100-page ML Book- Unfortunately almost no content about model assumptions or how to test for them.
| improve this question | | | | |
$\endgroup$
2
+50
$\begingroup$

Great question @Maeaex1 !

First of all why do we even need assumptions in models (generally speaking )?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources




But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

enter image description here

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

So to conclude: there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Well deserved bounty! Thanks for the answer. $\endgroup$ – Maeaex1 Dec 17 '19 at 16:29
1
$\begingroup$

I would like to comment, but i can't with my reputation. So here is an [article] (https://arxiv.org/pdf/1802.04474.pdf) with some assumption. You can treat neural network in the way of this article (non parametric estimation) and do the same assumption than this field (where there is few assumption, except about regularity of the distribution).

Hope it helps.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thanks for your response PauZen! I will definitely check the paper. If you add a short summary of the paper to your answer, I will accept your answer and the bounty is yours (if you are even interested :-D). Cheers! $\endgroup$ – Maeaex1 Dec 17 '19 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.