0
$\begingroup$

Upon Googling "Maxpool ReLU order" or similar, I've found many people saying this order does not effect the result, i.e.:

MaxPool(Relu(x)) = Relu(MaxPool(x))

Here are a small number of examples of people saying this:
https://stackoverflow.com/questions/35543428/activation-function-after-pooling-layer-or-convolutional-layer
https://github.com/tensorflow/tensorflow/issues/3180
https://www.quora.com/In-most-papers-I-read-the-CNN-order-is-convolution-relu-max-pooling-So-can-I-change-the-order-to-become-convolution-max-pooling-relu
https://towardsdatascience.com/convolution-neural-networks-a-beginners-guide-implementing-a-mnist-hand-written-digit-8aa60330d022

To be clear, I'm completely aware that there could be a slight speed difference, but what I'm asking about here is the computation result, not the speed.

For example, consider the following: enter image description here

How can the general consensus be that the ReLU/MaxPool order does not effect the computation result when it's easy to come up with a quick example where is does appear to effect the computation result?

For what it's worth, ChatGPT seems to go against the general consensus: enter image description here

$\endgroup$

1 Answer 1

4
$\begingroup$
  1. ChatGPT often speaks bullshit. Do not rely on it.
  2. Your example is wrong. On the second computation, you are computing the absolute value, not ReLU: ReLU(-5) = 0 and ReLU(-3) = 0. The result is 2, which is the same as the first computation.
  3. $\max(ReLU(x_1,...,x_n)) = ReLU(\max(x_1,...,x_n))$
$\endgroup$
1
  • 1
    $\begingroup$ good eyes on that abs() $\endgroup$ May 20, 2023 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.