I want to create sparse feed-forward networks in Pytorch and Tensorflow, i.e., say each node is only connected to k number of neurons of the next layer where k is strictly less than the total number of nodes in the next layer. But all the tutorials/examples I have seen so far are for fully connected feed-forward networks. Is there any way to construct such sparse networks in these tools?
We don't use sparse connections in feedforward to make use of an efficient matrix product operations.
But If you want to model that aspect, see dropout
Agreed with Fadi. It is just not very efficient. You have to split up the input and weights tensor, into tensor.size()/k chunks and then do a separate mult,add operation on each pair of chunks. Even if you had an efficient indexing scheme of concatenating the chunks to mul-add in one go, you would then have to un-index output to a flat batched tensor.
A few things to think about would be pathnet - pytorch and this (I have a half-working implementation somewhere if you are interested), but both are variations on the efficient version of what you are talking about, which is some form of routing.
Do you need specific edges or just a set sparsity level that doesn't change? I know Keras allows you to pass a random seed to the dropout layers. If I understand what you need correctly, you could just set the dropout percentage to get the sparsity you want and then set the random seed to always be the same so the layers always drop (and keep) the same connections.
This tutorial might be the way to go with tensorflow. The tree structure leads to sparsity between layers and it is explicitly programmed. Both convolutional layers and fully connected layers are used. The author shares links to the full code.