Take the following tensor: $$ \left[\begin{array}{cc} a & b & c\\ d & e & f\\ g & h & i\\ \end{array}\right] $$ $$ \left[\begin{array}{cc} j & k & l\\ o & n & m\\ p & q & r\\ \end{array}\right] $$

Where each matrix represents a channel.

This could be reshaped fairly easily into a vector: $$ [a,d,g,b,e,h,c,f,i,j,o,p,k,n,q,l,m,r] $$

And then concatenated row-wise with other vectorized tensors to form a typical flat-file dataset of dimension $N \times P$, where $N$ is the number of training samples and $P$ is the product of the tensor dimensions.

Rather than futzing with a convolutional layer, one could simply constrain ones weights to be zero in the subsequent layer.

If $X$ is a flat $N\times P$ dataset of concatenated vectorized tensors, then the convolutional weights would form a sparse matrix, with the first two columns of a $P \times 4$ convolutional "layer" being $2\times 2\times 2$ filter being $$ \left[\begin{array}{c} w 0\\ w0\\ 00\\ ww\\ ww\\ 00\\ 0w\\0w\\00\\w0\\ w0\\00\\ww\\ww\\0w\\0w\\0w\\00 \end{array}\right] $$

This seems to me more intuitive than the tensor formulation, and could be computed fairly simply using sparse matrix packages. Perhaps it is partly matter of taste. But I'm curious: is there anything special about the tensor paradigm -- either mathematically or computationally -- that is superior to the flattened representation? I understand that computers convert matrix algebra to for-loops "under the hood", but doesn't the advent of the GPU make such explicit looping irrelevant?

  • 1
    $\begingroup$ You are absolutely correct, however, to most people, it is more intuitive to understand them the way @Maxim has so well shown in his answer. You are using sparse matrices in order not to use tensors, and if you prefer to think of them the way you do, you should go ahead. $\endgroup$
    – user
    Oct 16, 2017 at 2:05

1 Answer 1


Tensors come pretty natural in convolutionals networks.

  • Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This information gets lost when you reshape an image into a vector. Look how a convolutional layer works.

conv layer

  • Training usually is done in batches, which is another dimension as far as a neural network is concerned. You don't want to mix different images into a matrix, they are totally independent.

  • Finally, there is depth channel. Initially it's one of R, G, B channels of the input image, and then each channel corresponds to a filter applied to the previous conv layer. Once again, filters are independent, it doesn't make sense to mix them up until the final layer.

So, in total there are 4-rank tensors going through the conv net. It's not only more intuitive (each dimension has a meaning), but also results in higher accuracy, because it employs all meaningful information from the images.

  • $\begingroup$ Any tensor can be represented in a lower-order space, and the topology of the weights can be represented by a specific pattern of matrix sparsity. I totally understand the intuition of a tensor, I just don't get why we are throwing away all of our matrix tools in computing the actual models that way. $\endgroup$ Oct 2, 2017 at 20:47
  • $\begingroup$ I'm sure I understand your point that "we are throwing away all of our matrix tools". Dot-product is the most frequent action during both forward and backward passes, and it can be done with tensors. I just don't see what additional tools you will get if you reduce the rank to 2. $\endgroup$
    – Maxim
    Oct 2, 2017 at 20:52
  • $\begingroup$ Rather than looping over the indices, the forward and backward passes are a single dot product to predict and the calculate the gradient, respectively. I guess I was being a bit hyperbolic in saying "throwing away our matrix tools". It just seems to me that the tensor representation is much harder to code for those who are accustomed to coding with matrices, and without much benefit given that a 2d representation is available ... unless I'm missing something. It is possible that it is all a matter of taste. But again, I'd like to be sure that I'm not missing something. $\endgroup$ Oct 2, 2017 at 21:00
  • $\begingroup$ I agree that work with tensors requires a bit more intuition and "fantasy" in some sense. Modern numerical libraries (I mostly worked with numpy and tensorflow) work with any rank by design. So with a fair amount of discipline tensor operations are coded just like matrices, just a matter of practice I guess. It certainly helps that dimensions have own meaning. $\endgroup$
    – Maxim
    Oct 2, 2017 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.