I want to create sparse feed-forward networks in Pytorch and Tensorflow, i.e., say each node is only connected to k number of neurons of the next layer where k is strictly less than the total number of nodes in the next layer. But all the tutorials/examples I have seen so far are for fully connected feed-forward networks. Is there any way to construct such sparse networks in these tools?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Have we found an efficient solution to this question? $\endgroup$– Pınar DemetciOct 12, 2018 at 16:19
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
2
Agreed with Fadi. It is just not very efficient. You have to split up the input and weights tensor, into tensor.size()/k chunks and then do a separate mult,add operation on each pair of chunks. Even if you had an efficient indexing scheme of concatenating the chunks to mul-add in one go, you would then have to un-index output to a flat batched tensor.
A few things to think about would be pathnet - pytorch and this (I have a half-working implementation somewhere if you are interested), but both are variations on the efficient version of what you are talking about, which is some form of routing.
-
$\begingroup$ as I mentioned in my previous comment to another answer, I need sparsity for some specific reason. I am looking for 'static sparsity', i.e., I just fix the sparsity of the network once, and then do the training on it. From a quick glance, the papers you showed seem to do something different that this? $\endgroup$– DaveMay 11, 2018 at 16:43
-
$\begingroup$ yes, they are different. I am thinking of 'paths' as the operations that split the inputs in a fixed way, and route the mul,adds to their targets. I figured the code may be adaptable from that to do your version of routing. It is definitely doable in pytorch, I just do not see an obvious way to do it efficiently. If it comes to me Ill post :) $\endgroup$ May 11, 2018 at 18:12
$\begingroup$
$\endgroup$
1
Do you need specific edges or just a set sparsity level that doesn't change? I know Keras allows you to pass a random seed to the dropout layers. If I understand what you need correctly, you could just set the dropout percentage to get the sparsity you want and then set the random seed to always be the same so the layers always drop (and keep) the same connections.
-
$\begingroup$ I need specific sparsity pattern and not random ones. $\endgroup$– DaveOct 13, 2018 at 18:53
$\begingroup$
$\endgroup$
1
This tutorial might be the way to go with tensorflow. The tree structure leads to sparsity between layers and it is explicitly programmed. Both convolutional layers and fully connected layers are used. The author shares links to the full code.
-
$\begingroup$ It is common to add also a short version of the link. Either as a summary or part of the main code. Links tend to become dead after some time and a user that will find this answer in the future, might not be able to check it. $\endgroup$– TasosApr 8, 2019 at 15:23
$\begingroup$
$\endgroup$
6
We don't use sparse connections in feedforward to make use of an efficient matrix product operations.
But If you want to model that aspect, see dropout
-
$\begingroup$ I know that the training may be tricky, but I have some specific reasons why I need sparsely connected network. I need static sparsity, i.e., once I fix the sparsity, I don't want to change it. This is, what I think you are alluding to through the dropout paper, different than randomly removing connections at every epoch etc. $\endgroup$– DaveMay 11, 2018 at 16:37
-
$\begingroup$ quote from tutorial on VAE "In general, we don’t need to worry about ensuring that the latent structure exists. If such latent structure helps the model accurately maximize the likelihood of the training set, then the network will learn that structure at some layer." And from DeepLearning book quote "The deep learning practitioner typically does not intend for the latent variables to take on any specific semantics ahead of time—the training algorithm is free to invent the concepts it needs to model a particular dataset" $\endgroup$ May 11, 2018 at 16:57
-
$\begingroup$ how does it help answering my specific question?! $\endgroup$– DaveMay 11, 2018 at 18:53
-
$\begingroup$ don't care too much about how to the latent variables should be represented because the network will figure it out. $\endgroup$ May 11, 2018 at 22:32
-
$\begingroup$ That's not the point. As I mentioned in my comment, I want to use sparse networks for some specific reason. $\endgroup$– DaveMay 12, 2018 at 0:33