Jan van der Vegt did a good job of mentioning some of the main irregular layer types, but I'll toss in a couple of others for the sake of completeness. I know from experience that layers in Teuvo Kohonen's Self-Organizing Maps can be fully connected, without requiring a square or rectangular matrix. I've gotten excellent results by creating topologies that match the underlying problems, which may feature jagged and possibly multidimensional shapes, then operating on them with sets instead of matrices. That's one reason I prefer using set-based languages like SQL to matrix math when coding neural nets. Another interesting neural net type worth exploring is Neural Gas, which expands across the space of data points to fit the natural topology of a problem, whatever that may be. It is easier to express in terms of manifolds than ordinary space. In a sense, it can be viewed as a generalization and abstraction of Kohonen nets.
Resources on Kohonen nets are easy to find, but the literature on Neural Gas is still scarce. Here are some worthwhile links I've read to Neural Gas and assorted variants:
• Martinetz and Schultern's original article on Neural Gas.
• The Wikipedia article on Neural Gas.
• A Master's Thesis on Growing Neural Gas, which dynamically creates the topology
• A Growing Neural Gas Network Learns Topologies, a research paper by Bernd Fritzke