I've come across the term "learnable parameters" recently, and googling didn't help much as most search was describing learnable parameters in a CNN instead of a DNN. Is there any difference between the two?
How would I compute the number of learnable parameters in a DNN? Could anyone please explain what those are with an example? I'm new to machine learning so I would appreciate some help on this.
Imagine a simple input vector $\boldsymbol{x}=[x_1,x_2,x_3]^T$, with $x_i\in \{0,1,\ldots,255\}$. If our neural network has a layer which normalizes the inputs into the range of rational numbers between $0$ and $1$. The output of this operation would be
$$\boldsymbol{x}_\text{Norm}=[1/255, 1/255, 1/255]\boldsymbol{x}.$$
As you can see we have $3$ parameters but all are not trainable.
A more common situation is given when you do transfer learning with pretrained neural networks. In this situation, you take a trained neural network and modify it (e.g. add additional layers or chop of some layers and add your own layers). Then when you want to train the new network your data set you can decide if you want to use the pretrained parameters (non-trainable parameters) or train the network from the scratch. In the last case, you will have a model in which all parameters are trainable.
In neural networks in general, and in deep learning algorithms (CNN, DNN, etc.) that are also based on neural networks, learnable parameters are parameters that will be learned by the model during the training procedure such as weights and biases.
You can generate learnable parameters for each layer of your model.
DNN is a deep neural network. What I understand that when a neural network becomes deep, it can be said that it's a deep learning model.
So, to compute a DNN or CNN learnable parameters, you can build your model using Keras. It can automatically generate the models learnable parameters for each layer. When you complete building your model, type model.summary(). It generates a list of learnable parameters when you run it.
This link builds a DNN model using using Keras.
def count_parameters(model): return sum(p.numel() for p in model.parameters() if p.requires_grad) print(f'The model has {count_parameters(model):,} trainable parameters')and in Keras you have model.summary() $\endgroup$