2
$\begingroup$

I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.

enter image description here

The second image represents the pyramid pooling module which includes several scale images.

enter image description here

I am looking forward to the addition and multiplication function's attribute in a deep neural network.

The question is:

Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?

|improve this question
$\endgroup$
  • $\begingroup$ Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so. $\endgroup$ – D.W. Feb 16 at 21:46
  • $\begingroup$ I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for. $\endgroup$ – Sean Owen Feb 17 at 0:56
  • $\begingroup$ dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again? $\endgroup$ – amir Maleki Feb 17 at 8:14
  • $\begingroup$ Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function. $\endgroup$ – amir Maleki Feb 17 at 8:26
  • 1
    $\begingroup$ What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them. $\endgroup$ – Sean Owen Feb 17 at 15:36
0
$\begingroup$

The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .

|improve this answer
$\endgroup$
  • $\begingroup$ dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here. $\endgroup$ – amir Maleki Feb 26 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.