Reasoning behind using Deep Learning on non-local data

Question

I understand the using of deep learning for data that have "local" structure, for example, images/videos/texts, as the convolutional layers reduce the amount of dimensions.

However, I saw that some people use it on non-local data, as on databases for example, here or here on the titanic database.

My question is: as just one hidden layer with enough neurons within can theoretically creates as many dimensions as we want, why would one use several hidden layers/deep learning instead of just using a single bigger hidden layer?

Answer 1

Convnets and dependency

Convnets are structured in a way that spatial dependency is encoded in the model. They do not explicitly "reduce amount of dimensions".

This means that while a fully contacted layer will consider relationships between all features, a convolutional layer will assume that neighbouring features are connected between them and whereas distant features are not dependent with each other.

We could feed an image into a fully connected layer by lining up all pixels one under the other in a huge column and pass it to the neural network. However, that would be a very expensive and redundant computation, given we know that "links" between distant features have a very low probability of being inter-dependent.

Network depth and summarisation

As correctly stated in your question, a feed-forward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function an arbitrary precision (Universal approximation theorem).

But, let's have a look at what a single hidden layer cannot do.

1. Non-linear modelling $\neq$ non-linear relationships

Non-linear transformations or curvature (e.g. polynomial regression, logistic unit etc.) is often misread as non-linearity in model parameters (non-linear models).

As an example, in the following neural network, the output of the first layer neuron would be:

$$a_1=f(𝑤𝑥+𝑏) \ (1)$$

If our activation function $f(x)$ is linear, this is in essence linear regression:

$$a_1 = wx + b\ (2)$$

If we added another layer with one neuron as in the image below and with linear activations, the output would be:

$$a_1 = w_1x + b_1 \ (3), \ (first\ layer)$$

$$𝑎_2 = 𝑤_2a_1+𝑏2 \ (4), \ (second\ layer)$$

This reduces to,

$$a_2 = (w_2*w_1)x + w_2*b_1 + b_2\ (5)$$

The $(w_2*w_1)$ and $w_2*b_1$ terms in essence that the model is "non-linear in its parameters" but still "linear in the variables ($x$). In essence, despite linear activation functions the 2-layer neural network is a nonlinear model of a linear relationship. Non-linearity in both the variables and parameters are important because you cannot replicate this two layer neural network with a single regression model and capture all effects of the model.

• Activation function (per layer): if non-linear, introduces non-linear relationships in the variables. Helpful for altering and/or expanding the solution space.
• Network depth (manier layers/connections): introduce non-linearities in the parameters which are necessary for non-degenerate solutions going beyond regression. In essence, the use of multiple hidden layers allows construction of hierarchical features at different levels of resolution [1]

2. Parametric nonlinear regression $\neq$ neural network

On a similar point, it is common to depict neural networks as parametric nonlinear regression, multiple regression etc. Deep neural networks are a particular beast when the two different elements of nonlinearities as described above are combined;

1. DNNs allow fitting models for an insane amount of parameters, impossible to fit with a Levenberg-Marquard for say nonlinear least squares.
2. Although theoretically "parametric", in practise very few (if any) hyperparameters of a DNN are fixed a-prior and in fact the entirety of its modelling pipeline (from preprocessing to architecture and regularisation) are part of model tuning which usually results in fitting a model from a much bigger class of models.
3. DNNs scale to huge datasets.

Anyway, hope this helps.

See also:\

1. Elements of Statistical learning
2. https://stats.stackexchange.com/a/33891/110383
3. https://stackoverflow.com/a/61619406/11545502
4. https://stats.stackexchange.com/a/345065/110383

Answer 2

It appear that I forgot one point of the ANN, at least forgot one of its effects : the activation function.

It is true that for linear activation, multi-layer can be reduce to a single-one, but with a non-linear function,

a two-layer neural network can be proven to be a universal function approximator.

Sources

However, it is true that I dont understand now why to use more than two hidden layers...

Answer 3

A couple answers:

Yes, it could be overkill in scenarios where simpler models suffice. Linear and logistic regression are trivially also representable as a neural network, but it's not the most efficient way to solve it.

On the plus side, deep learning frameworks are good at applying specialized hardware like GPUs. Where a problem also fits deep learning, it could be a performance win if GPUs are available.

It can learn non-linear relationships via the activation functions. That doesn't mean it easily learns, say, interaction features. Yes it's possible to approximate anything with two wide enough dense layers, but they would have to be ridiculously wide to learn some arbitrary functions.

They're useful for timeseries data, but that is kind of data with a 'locality' in a time dimension, which you're already ruling in.

The intermediate representation could be meaningful for other purposes. For example a network that learns to classify purchase intent from customer attributes produces an intermediate representation that might more meaningfully yield to clustering than the raw input. The embedding captures the input in a space that is meaningful with respect to the target.

Answer 4

A 1-hidden-layer neural network is a universal approximator, but adding more layers can give a more efficient representation.

A simple example: suppose you want to match the pattern ‘((A or B or C) and (D or E or F)) or G’ using simple on-off neurons.

• Using 2 hidden layers: neuron 1 in the first hidden layer (h1_1) recognizes (A or B or C), neuron 2 in the first hidden layer (h1_2) recognizes (D or E or F); then in the second layer h2_1 recognizes the simultaneous activation of h1_1 and h1_2; then the output is active when either h2_1 or G is active. This requires 3 hidden neurons.

• Using one hidden layer you have to distribute all the possible pattern matches to separate neurons in the first hidden layer. So h1_1 would recognize ‘ADG’, h1_2 would recognize ‘BDG’, etc. and the output neuron is active when any of the hidden neurons is active. This requires 9 hidden neurons.

Multilayer networks tend to be more efficient any time you have that sort of nested and-or logic. Image processing is just one of those cases.