Why torch activation functions contains parameters?

Question

What is the purpose of the weights?

Answer 1

There are no weights for the nn.Tanh module.

The extra weights you see are the bias weights for the linear layers.

This is clear if you look at model.state_dict(), which includes parameter keys defining where the weights go.

model = nn.Sequential(
                nn.Linear(1, 5),
                nn.Tanh(),
                nn.Linear(5,5),
                nn.Tanh(),
                nn.Linear(5, 1)
                )

print(model.state_dict())

> OrderedDict([('0.weight',
              tensor([[ 0.2226],
                      [-0.6180],
                      [ 0.1934],
                      [ 0.9877],
                      [-0.5451]])),
             ('0.bias', tensor([ 0.0996,  0.9742,  0.3510, -0.2562, -0.5217])),
             ('2.weight',
              tensor([[ 0.1977, -0.2723,  0.2607,  0.0615, -0.4093],
                      [ 0.0772, -0.4179, -0.2974,  0.2643, -0.4437],
                      [ 0.3902, -0.4201, -0.1676,  0.0753,  0.2992],
                      [-0.1437,  0.4166,  0.0059,  0.2098, -0.1795],
                      [ 0.0254,  0.0849,  0.0433, -0.0336, -0.2402]])),
             ('2.bias', tensor([ 0.0929,  0.2627, -0.2258, -0.1396, -0.2986])),
             ('4.weight',
              tensor([[ 0.0866,  0.3795,  0.0632,  0.0361, -0.4052]])),
             ('4.bias', tensor([0.1559]))])
```