# How are ANN's, RNN's related to logistic regression and CRF's?

This question is about placing the classes of neural networks in perspective to other models.

In "An Introduction to Conditional Random Fields" by Sutton and McCallum, the following figure is presented:

It shows that Naive Bayes and Logistic Regression form a generative/discriminative pair and that linear-chain CRFs are a natural extension of logistic regression to sequences.

My question: is it possible to extend this figure to also contain (certain kinds) of neural networks? For example, a plain feedforward neural network can be seen as multiple stacked layers of logistic regressions with activation functions. Can we then say that linear-chain CRF's in this class are a specific kind of Recurrent Neural Networks (RNN's)?

These graphs (Probabilistic Graphical Models, PGMs) represent random variables $$X=\{X_1,X_2,..,X_n\}$$ (circles) and their statistical dependence (lines or arrows). They together define a structure for joint distribution $$P_X(\boldsymbol{x})$$; i.e., a PGM factorizes $$P_X(\boldsymbol{x})$$. As a reminder, each data point $$\boldsymbol{x}=(x_1,..,x_n)$$ is a sample from $$P_X$$. However, a neural network represents computational units (circles) and flow of data (arrows). For example, node $$x_1$$ connected to node $$y$$ with weight $$w_1$$ could mean $$y=\sigma(w_1x_1+...)$$. They together define a function $$f(\boldsymbol{x};W)$$.
To illustrate PGMs, suppose random variables $$X_1$$ and $$X_2$$ are features and $$C$$ is label. A data point $$\boldsymbol{x}=(x_1, x_2, c)$$ is a sample from distribution $$P_X$$. Naive Bayes assumes features $$X_1$$ and $$X_2$$ are statistically independent given label $$C$$, thus factorizes $$P_X(x_1, x_2, c)$$ as $$P(x_1|c)P(x_2|c)P(c)$$.
In some occasions, neural networks and PGMs can become related, although not through their circle-line representations. For example, neural networks can be used to approximate some factors of $$P_X(\boldsymbol{x})$$ like $$P_X(x_1|c)$$ with function $$f(x_1,c;W)$$. As another example, we can treat weights of a neural network as random variables and define a PGM over weights.