How does tensor product/multiplication work in TensorFlow?

Question

In Tensorflow, I saw the following example:

import tensorflow as tf 
import numpy as np 

mat_a = tf.constant(np.arange(1,13, dtype=np.int32), shape=[2,2,3])  
mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])  
mul_c = tf.matmul(mat_a, mat_b)

with tf.Session() as sess:  
   runop = sess.run(mul_c)  
   print(runop) 

[[[ 88  94]  
  [214 229]]  
 [[484 508]  
  [642 674]]]

How does the tensor multiplication work?

Answer 1

You may want to read the documentation.

output[..., i, j] = sum_k (a[..., i, k] * b[..., k, j]), for all indices i, j.

For instance, in your example

$~~88=1\times12+2\times14+3\times16,~~~94=1\times13+2\times15+3\times17$ $214=4\times12+5\times14+6\times16,~229=4\times13+5\times15+6\times17$

Answer 2

I'll give you a small example, if you do the following Kronecker product \begin{equation} \begin{bmatrix} \color{red}{1} \\ \color{green}{5} \\ \color{blue}{10} \end{bmatrix} \otimes \begin{bmatrix} 2 \\ 4 \end{bmatrix} = \begin{bmatrix} \color{red}{1} \begin{bmatrix} 2 \\ 4 \end{bmatrix} \\\\ \color{green}{5} \begin{bmatrix} 2 \\ 4 \end{bmatrix} \\\\ \color{blue}{10} \begin{bmatrix} 2 \\ 4 \end{bmatrix} \\ \end{bmatrix} = \begin{bmatrix} 2 \\ 4 \\ 10 \\ 20 \\ 20 \\ 40 \end{bmatrix} \end{equation} The Kronecker product works the same way for matrices as well.

Answer 3

Tensor multiplication is just a generalization of matrix multiplication which is just a generalization of vector multiplication.

Matrix multiplication is defined as:

$$ A_i \cdot B_j = C_{i, j}$$

where $i$ is the $i^{th}$ row, $j$ is the $j^{th}$ column, and $\cdot$ is the dot product. Therefore it just a series of dot products.

One can then see how this extends to tensors: $$\mathbf{A}_{i} \cdot \mathbf{B}_{j} = \mathbf{C}_{i, j} $$

where $i$ is the $i^{th}$ row-wise matrix of the tensor, and $j$ is the $j^{th}$ column-wise matrix of the tensor... and is therefore just a series of matrix multiplications - or a series of a series of dot products.

Assuming all tensors are of rank three(it can be described with three coordinates):

$$\mathbf{A} \otimes \mathbf{B} = \mathbf{A}_{i, j} \cdot \mathbf{B}_{j, k} = \mathbf{C}_{i, j, k}$$

which means the $(i,j)^{th}$ vector of $\mathbf{A}$ times the $(j, k)^{th}$ vector of $\mathbf{B}$.