This question is about placing the classes of neural networks in perspective to other models.

In "An Introduction to Conditional Random Fields" by Sutton and McCallum, the following figure is presented:

enter image description here

It shows that Naive Bayes and Logistic Regression form a generative/discriminative pair and that linear-chain CRFs are a natural extension of logistic regression to sequences.

My question: is it possible to extend this figure to also contain (certain kinds) of neural networks? For example, a plain feedforward neural network can be seen as multiple stacked layers of logistic regressions with activation functions. Can we then say that linear-chain CRF's in this class are a specific kind of Recurrent Neural Networks (RNN's)?


No, it is not possible.

There is a fundamental difference between these graphs and neural networks, despite both being represented by circles and lines/arrows.

These graphs (Probabilistic Graphical Models, PGMs) represent random variables $X=\{X_1,X_2,..,X_n\}$ (circles) and their statistical dependence (lines or arrows). They together define a structure for joint distribution $P_X(\boldsymbol{x})$; i.e., a PGM factorizes $P_X(\boldsymbol{x})$. As a reminder, each data point $\boldsymbol{x}=(x_1,..,x_n)$ is a sample from $P_X$. However, a neural network represents computational units (circles) and flow of data (arrows). For example, node $x_1$ connected to node $y$ with weight $w_1$ could mean $y=\sigma(w_1x_1+...)$. They together define a function $f(\boldsymbol{x};W)$.

To illustrate PGMs, suppose random variables $X_1$ and $X_2$ are features and $C$ is label. A data point $\boldsymbol{x}=(x_1, x_2, c)$ is a sample from distribution $P_X$. Naive Bayes assumes features $X_1$ and $X_2$ are statistically independent given label $C$, thus factorizes $P_X(x_1, x_2, c)$ as $P(x_1|c)P(x_2|c)P(c)$.

In some occasions, neural networks and PGMs can become related, although not through their circle-line representations. For example, neural networks can be used to approximate some factors of $P_X(\boldsymbol{x})$ like $P_X(x_1|c)$ with function $f(x_1,c;W)$. As another example, we can treat weights of a neural network as random variables and define a PGM over weights.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.