What are differentiable modules used in deep learning

Question

I am reading this paper.

Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spa- tial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the abil- ity to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, result- ing in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.

Spatial Transformers are used in CNNs to have spatial invariant transformations and consequently the process of learning would be so easier and the networks would have better performance for data with different kind of distributions (noisy data). In this paper I don't realize the meaning of differentiable module. These so called differentiable modules are used in neural networks. But what is the meaning of differentiable?

Answer 1

"Differentiable" means that you can compute the derivative of the operations in the module, and therefore you can compute the gradients of the loss function with respect to the module parameters (i.e. use backpropagation).

This is normally a requirement for operations involved in neural network computations.

Note: you can use non-differentiable operations as part of a computational graph, but you won't be able to backpropagate gradients through them, and therefore any learnable parameters involved in operations prior to the non-differentiable one would not be able to learn.