Are there more layer types like convolution layers and fully connected layers?

Question

I've been learning about neural networks and I'm curious: Are there any other layer types like convolution and fully connected layers? I know there are things like max pooling or various regularization layers, but I'm mainly interested in more variations on fully connected layers (like how convolution layers do the same thing as fully connected layers but take the shape of data into account). It just seems strange that we wouldn't need anything except straight matrix multiplication or matrix multiplication on smaller chunks.

Do these exist? If not, why not?

Answer 1

Jan van der Vegt did a good job of mentioning some of the main irregular layer types, but I'll toss in a couple of others for the sake of completeness. I know from experience that layers in Teuvo Kohonen's Self-Organizing Maps can be fully connected, without requiring a square or rectangular matrix. I've gotten excellent results by creating topologies that match the underlying problems, which may feature jagged and possibly multidimensional shapes, then operating on them with sets instead of matrices. That's one reason I prefer using set-based languages like SQL to matrix math when coding neural nets. Another interesting neural net type worth exploring is Neural Gas, which expands across the space of data points to fit the natural topology of a problem, whatever that may be. It is easier to express in terms of manifolds than ordinary space. In a sense, it can be viewed as a generalization and abstraction of Kohonen nets.

Resources on Kohonen nets are easy to find, but the literature on Neural Gas is still scarce. Here are some worthwhile links I've read to Neural Gas and assorted variants:

• Martinetz and Schultern's original article on Neural Gas.

• The Wikipedia article on Neural Gas.

• A Master's Thesis on Growing Neural Gas, which dynamically creates the topology

• A Growing Neural Gas Network Learns Topologies, a research paper by Bernd Fritzke

Answer 2

Recurrent neural networks have different types of layers, Long Short Term Memory layers for example. Neural Turing Machines are even wilder. Another example is a non-square convolution layer, I was looking at writing a CNN reinforcement learning program that could learn how to play Abalone, which is played on a hexagon board, where similar patterns appear over the board in hexagon patches, so that is also a possibility.